I was designing a file server using socket programming in C. I send the calls like open(), write() etc as plain strings using stream sockets and decipher it at the server end.i.e if it is a open call then we extract path, mode, flags. Is it ok or should I be using some kind of struct to store the file system calls and send to server where the server simply accesses the fields.
Is there some standard way i don't know?
Thanks
You're basically starting to define your own protocol. It would be a lot easier if you sent numbers describing operations instead of strings.
If you're serious about this you might want to look into RPC - RFC707 (you did ask for a standard way, right?).
Yes, there is a standard way. Look into NFS, AFP, CIFS, and WebDAV, then pick one.
You already have answers for the standard way, so I'll give you a few caveats you should look out for.
If you intend to deploy your file server in an un-trusted environment (e.g. on the Internet) think about securing it right away. Securing it is not just a question of slapping encryption on - you need to know how you intend to authenticate your users, how you want to authorize different types of access to different parts of the server, how you will insure the authenticity and the integrity of the data and how you intend to keep the data confidential.
You'll also need to think about your server's availability. That means that you should be fault-tolerant - i.e. connections can (and will) break (regardless of whether they're being broken on purpose or not) so you need to detect that, either will some kind of keep-alive (which will fail if the client left) or with some kind of activity time-out (which will expire if the client left). You also need to think about how many clients you are willing to support simultaneously - which can radically change the architecture of your server.
As for the open, close, read, write etc. commands, most file transfer protocols don't go into so much detail, but it may be interesting to be able to do so depending on your situation. If your files are huge and you only need some chunks of it, or if you want to be able to lock files to work on them exclusively, etc. you may want to go into such detail. If you don't have those requirements, simpler, transactional commands such as get & put (rather than open, read, read, read, close and open, write, write some more, close) may both be easier to implement and easier to work with.
If you want a human being to interact with your server and give it commands, text is a good approach: it's easy to debug when sniffing and humans understand text and can type is easily. If there are no humans involved, using integers as commands is probably a better approach: you can structure your command to start with an integer followed by a number of parameters and always simply expect the same thing on the server's end (and do a switch on the command you receive). Even in that case, though, it may be a good idea to have human-readable values in your integers. For example, putting 'READ' in an integer as the read command uses as many bytes as 0x00000001, but is easier to read when sniffed with WireShark.
Finally, you really take a look at the standard approaches and try to understand the trade-offs made in each case. Ask yourself, for example, why HTTP has such verbose headers and why WebDAV uses it. Why does FTP use two connections (one for commands and one for data) while many other protocols use only one? How did NFS evolve to where it is now, and why? Understanding the answers to these questions will help you develop your own protocol - if after you understand those answers, you still feel you need your own protocol.
Related
I am asked for an assignment that requires implementation of FTP protocol. I have gone through the documentation given at RFC959.
I am confused with a couple of implementation details
1)If a file needs to be transferred, what function can be used. can a simple send() function be used for a non text file.
2) Is it possible to get a good tutorial that speaks about implementing Modes and file structures, and to specify, which are essential.
hope to get a reply soon.
FTP transfers file through a plain TCP connection, and you can transfer any kind of file with it. There is no difference between text files and binary files, they are all just sequence of bytes.
For the file transmission is sufficient to open a connection and call the write function many times until the entire file is transmitted (check the return value of write to know how many bytes it sent).
The rest of the FTP protocol is text based and is sent to a different port.
There is a good tutorial on using FTP directly through netcat, that can be useful to understand how things work. Understanding active and passive mode can also be useful, since you are going to implement at least one of them.
Also, use wireshark to follow a TCP stream and see the data you are sending/receiving, it can be very useful in debugging.
The protocol implementation won't give you a file structure. The protocol is here to define some rules and states.
The dev/prog part is up to you. You just need to respect the FTP protocol in order to gain the normalization and the compatibility with other client/server.
Best regards
I am trying to write a client-server program in C in windows. The objective is to receive the directory listing from the server. Now I was trying to develop the client-server in such a way to utilize most resources.
One way to implement is that server makes a single send() call to send info of a single file. So if there are 100 files, it makes 100 calls. But I feel its a wastage of network resources. As far as I know the buffer size for send() or recv() in windows is 8kb. But the info of a single file will be hardly 1kb. So is there a way to make send() call to send multiple files info (file info are stored in structures. So they basically form a linked list) ? May be I can send info of atleast 8 files in a single Send() call. That should reduce the total send() calls to maximum 13.
So basically is there a way to send a linked list via send() ?? Plz let me know if you can think of any alternative method.
Good question! +1 for that.
But do you really want or need to write your code to use Winsock? There are good reasons to do so -- including that it's fun and a challenge. But if you don't need to, you might want to consider using the libcurl ftp library, which is free, multi-platform (including win32, of course), just works, and might make your job a lot easier.
The only way I know of to do this with FTP is to use multiple connections to the FTP server. If this is allowed by the server, there can be a list performance boost because the many protocol exchanges needed to list a complete folder tree can be run in parallel.
Rgds,
Martin
TCP is a byte stream. There is no guarantee of a 1-to-1 relation between the number of items you want to send and the number of calls to send() (or recv()) you need to make. That is simply not how TCP works. You format the data the way you need to, and then you keep calling send() until it tells you that all of the data has been sent.
Regarding FTP, please read RFC 959 and RFC 3659 to learn how the ftp protocol actually works. Before the introduction of the MLST and MLSD commands, directory listings had no standardized format. FTP servers were free to use whatever formatting they wanted. Many servers just piped the raw data from the OS's own dir or list commands. Indy, for example, includes several dozen parsers in its FTP client for handling non-standard directory listings.
I am developing some experimental setup in C.
I am exploring a scenario as follows and I need help to understand it.
I have a system A which has a lot of Applications using cryptographic algorithms.
But these crypto calls(openssl calls) should be sent to another System B which takes care of cryptography.
Therefore, I have to send any calls to cryptographic (openssl) engines via socket to a remote system(B) which has openssl support.
My plan is to have a small socket prog on System A which forwards these calls to system B.
What I'm still unclear at this moment is how I handle the received commands at System B.
Do I actually get these commands and translate them into corresponding calls to openssl locally in my system? This means I have to program whatever is done on System A right?
Or is there a way to tunnel/send these raw lines of code to the openssl libs directly and just received the result and then resend to System A
How do you think I should go about the problem?
PS: Oh by the way, the calls to cryptography(like EngineUpdate, VerifyFinal etc or Digest on System A can be either on Java or C.. I already wrote a Java/C program to send these commands to System B via sockets...
The problem is only on System B and how I have to handle..
You could use sockets on B, but that means you need to define a protocol for that. Or you use RPC (remote procedure calls).
Examples for socket programming can be found here.
RPC is explained here.
The easiest (not to say "the easy", but still) way I can imagine would be to:
Write wrapper (proxy) versions of the libraries you want to make remote.
Write a server program that listens to calls, performs them using the real local libraries, and sends the result back.
Preload the proxy library before running any application where you want to do this.
Of course, there are many many problems with this approach:
It's not exactly trivial to define a serializing protocol for generic C function calls.
It's not exactly trivial to write the server, either.
Applications will slow a lot, since the proxy call needs to be synchronous.
What about security of the data on the network?
UPDATE:
As requested in a comment, I'll try to expand a bit. By "wrapper" I mean a new library, that has the same API as another one, but does not in fact contain the same code. Instead, the wrapper library will contain code to serialize the arguments, call the server, wait for a response, de-serialize the result(s), and present them to the calling program as if nothing happened.
Since this involves a lot of tedious, repetitive and error-prone code, it's probably best to abstract it by making it code-driven. The best would be to use the original library's header file to define the serialization needed, but that (of course) requires quite heavy C parsing. Failing that, you might start bottom-up and make a custom language to describe the calls, and then use that to generate the serialization, de-serialization, and proxy code.
On Linux systems, you can control the dynamic linker so that it loads your proxy library instead of the "real" library. You could of course also replace (on disk) the real library with the proxy, but that will break all applications that use it if the server is not working, which seems very risky.
So you basically have two choices, each outlined by unwind and ammoQ respectively:
(1) Write a server and do the socket/protocol work etc., yourself. You can minimize some of the pain by using solutions like Google's protocol buffers.
(2) use an existing middleware solution like (a) message queues or (b) an RPC mechanism like CORBA and its many alternatives
Either is probably more work than you anticipated. So really you have to answer this yourself. How serious is your project? How varied is your hardware? How likely is the hardware and software configuration to change in the future?
If this is more than a learning or pet project you are going to be bored with in a month or two then an existing middleware solution is probably the way to go. The downside is there is a somewhat intimidating learning curve.
You can go the RPC route with CORBA, ICE, or whatever the Java solutions are these days (RMI? EJB?), and a bunch of others. This is an elegant solution since your calls to the remote encryption machine appear to your SystemA as simple function calls and the middleware handles the data issues and sockets. But you aren't going to learn them in a weekend.
Personally I would look to see if a message queue solution like AMQP would work for you first. There is less of a learning curve than RPC.
Most of the applications I've seen that use TCP, do roughly the following to connect to remote host:
get the hostname (or address) from the configuration/user input (textual)
either resolve the hostname into address and add the port, or use getaddrinfo()
from the above fill in the sockaddr_* structure with one of the remote addresses
use the connect() to get the socket connected to the remote host.
if fails, possibly go to (3) and retry - or just complain about the error
(2) is blocking in the stock library implementation, and the (4) seems to be most frequently non-blocking, which seems to give a room for a lot of somewhat similar yet different code that serves the purpose to asynchronously connect to a remote host by its hostname.
So the question: what are the good reasons not to have the additional single call like following:
int sockfd = connect_by_name(const char *hostname, const char *servicename)
?
I can come up with three:
historic: because that's what the API is
provide for custom per-application policy mechanism for address selection/connection retry: this seems a bit superficial, since for the common case ("get me a tube to talk to remote host") the underlying OS should know better
provide the visual feedback to the user about the exact step involved ("name resolution" vs "connection attempt"): this seems rather important, lookup+connection attempt may take time
Only the last of them seems to be compelling enough to rewrite the resolve/connect code for every client app (as opposed to at least having and using a widely used library that would implement the connect_by_name() semantics in addition to the existing sockets API), so surely there should be some more reasons that I am missing ?
(one of the reasons behind the question is that this kind of API would appear to help the portability to IPv6, as well as possibly to other stream transport protocols significantly)
Or, maybe such a library exists and my google-fu failed me ?
(edited: corrected the definition to look like it was meant to look, thanks LnxPrgr3)
Implementing such an API with non-blocking characteristics within the constraints of the standard library (which, crucially, isn't supposed to start its own threads or processes to work asynchronously) would be problematic.
Both the name lookup and connecting part of the process require waiting for a remote response. If either of these are not to block, then that requires a way of doing asychronous work and signalling the change in state of the socket to the calling application. connect is able to do this, because the work of the connect call is done in the kernel, and the kernel can mark the socket as readable when the connect is done. However, name lookup is not able to do this, because the work of a name lookup is done in userspace - and without starting a new thread (which is verboten in the standard library), giving that name lookup code a way to be woken up to continue work is a difficult problem.
You could do it by having your proposed call return two file descriptors - one for the socket itself, and another that you are told "Do nothing with this file descriptor except to check regularly if it is readable. If this file descriptor becomes readable, call cbn_do_some_more_work(fd)". That is clearly a fairly uninspiring API!
The usual UNIX approach is to provide a set of simple, flexible tools, working on a small set of object types, that can be combined in order to produce complex effects. That applies to the programming API as much as it does to the standard shell tools.
Because you can build higher level APIs such as the one you propose on top of the native low level APIs.
The socket API is not just for TCP, but can also be used for other protocols that may have different end point conventions (i.e. the Unix-local protocol where you have a name only and no service). Or consider DNS which uses sockets to implement itself. How does the DNS code connect to the server if the connection code relies on DNS?
If you would like a higher level abstraction, one library to check out is ACE.
There are several questions in your question. For instance, why not
standardizing an API with such connect_by_name? That would certainly
be a good idea. It would not fit every purpose (see the DNS example
from R Samuel Klatchko) but for the typical network program, it would
be OK. A paper exploring such APIs is "Simplifying Internet Applications Development
With A Name-Oriented Sockets Interface" by Christian Vogt. Note
that another difficulty for such an API would be "callback"
applications, for instance a SIP client asking to be called back: the
application has no easy way to know its own name and therefore often
prefer to be called back by address, despite the problems it make, for
instance with NAT.
Now, another question is "Is it possible to build such
connect_by_name subroutine today?" Partly yes (with the caveats
mentioned by caf) but, if written in userspace, in an ordinary
library, it would not be completely "name-oriented" since the Unix
kernel still manages the connections using IP addresses. For instance,
I would expect a "real" connect_by_name routine to be able to
survive renumbering (for instance because a mobile host renumbered),
which is quite difficult to do in userspace.
Finally, yes, it already exists a lot of libraries with similar
semantics. For a HTTP client (the most common case for a program whose
network abilities are not the main feature, for instance a XML
processor), you have Neon and libcURL. With libcURL, you can
simply write things like:
#define URL "http://www.velib.paris.fr/service/stationdetails/42"
...
curl_easy_setopt(curl, CURLOPT_URL, URL);
result = curl_easy_perform(curl);
which is even higher-layer than connect_by_name since it uses an
URL, not a domain name.
The user, administrators and support staff need detailed runtime and monitoring information from a daemon developed in C.
In my case these information are e.g.
the current system health, like throughput (MB/s), already written data, ...
the current configuration
I would use JMX in the Java world and the procfs (or sysfs) interface for a kernel module. A log file doesn't seem to be the best way.
What is the best way for such a information interface for a C daemon?
I thought about opening a socket and implementing a bare-metal http or xmlrpc server, but that seems to be overkill. What are alternatives?
You can use a signal handler in your daemon that reacts to, say USR1, and dumps information to the screen/log/net. This way, you can just send the process a USR1 signal whenever you need the info.
You could listen on a UNIX-domain socket, and write regularly write the current status (say once a second) to anyone who connects to it. You don't need to implement a protocol like HTTP or XMLRPC - since the communication will be one-way just regularly write a single line of plain text containing the state.
If you are using a relational database anyway, create another table and fill it with the current status as frequent as necessary. If you don't have a relational database, write the status in a file, and implement some rotation scheme to avoid overwriting a file that somebody reads at that very moment.
Write to a file. Use a file locking protocol to force atomic reads and writes. Anything you agree on will work. There's probably a UUCP locking library floating around that you can use. In a previous life I found one for Linux. I've also implemented it from scratch. It's fairly trivial to do that too.
Check out the lockdev(3) library on Linux. It's for devices, but it may work for plain files too.
I like the socket idea best. There's no need to support HTTP or any RPC protocol. You can create a simple application specific protocol that returns requested information. If the server always returns the same info, then handling incoming requests is trivial, though the trivial approach may cause problems down the line if you ever want to expand on the possible queries. The main reason to use a pre-existing protocol is to leverage existing libraries and tools.
Speaking of leveraging, another option is to use SNMP and access the daemon as a managed component. If you need to query/manage the daemon remotely, this option has its advantages, but otherwise can turn out to be greater overkill than an HTTP server.