Related
There are several questions dealing with some aspects of this problem, but neither seems to answer it wholly. The whole problem can be summarized as follows:
You have an already compiled executable (obviously expecting the use of this technique).
You want to add an arbitrarily sized binary data to it (not necessarily by itself which would be another nasty problem to deal with).
You want the already compiled executable to be able to access this added binary data.
My particular use-case would be an interpreter, where I would like to make the user able to produce a single file executable out of an interpreter binary and the code he supplies (the interpreter binary being the executable which would have to be patched with the user supplied code as binary data).
A similar case are self-extracting archives, where a program (the archiving utility, such as zip) is capable to construct such an executable which contains a pre-built decompressor (the already compiled executable), and user-supplied data (the contents of the archive). Obviously no compiler or linker is involved in this process (Thanks, Mathias for the note and pointing out 7-zip).
Using existing questions a particular path of solution shows along the following examples:
appending data to an exe - This deals with the aspect of adding arbitrary data to arbitrary exes, without covering how to actually access it (basically simple append usually works, also true with Unix's ELF format).
Finding current executable's path without /proc/self/exe - In companion with the above, this would allow getting a file name to use for opening the exe, to access the added data. There are many more of these kind of questions, however neither focuses especially on the problem of getting a path suitable for the purpose of actually getting the binary opened as a file (which goal alone might (?) be easier to accomplish - truly you don't even need the path, just the binary opened for reading).
There also may be other, probably more elegant ways around this problem than padding the binary and opening the file for reading it in. For example could the executable be made so that it becomes rather trivial to patch it later with the arbitrarily sized data so it appears "within" it being in some proper data segment? (I couldn't really find anything on this, for fixed size data it should be trivial though unless the executable has some hash)
Can this be done reasonably well with as little deviation from standard C as possible? Even more or less cross-platform? (At least from maintenance standpoint) Note that it would be preferred if the program performing the adding of the binary data didn't rely on compiler tools to do it (which the user might not have), but solutions necessiting those might also be useful.
Note the already compiled executable criteria (the first point in the above list), which requires a completely different approach than solutions described in questions like C/C++ with GCC: Statically add resource files to executable/library or SDL embed image inside program executable , which ask for embedding data compile-time.
Additional notes:
The problems with the obvious approach outlined above and suggested in some comments, that to just append to the binary and use that, are as follows:
Opening the currently running program's binary doesn't seem something trivial (opening the executable for reading is, but not finding the path to supply to the file open call, at least not in a reasonably cross-platform manner).
The method of acquiring the path may provide an attack surface which probably wouldn't exist otherwise. This means that a potential attacker could trick the program to see different binary data (provided by him) like which the executable actually has, exposing any vulnerability which might reside in the parser of the data.
It depends on how you want other systems to see your binary.
Digital signed in Windows
The exe format allows for verifying the file has not been modified since publishing. This would allow you to :-
Compile your file
Add your data packet
Sign your file and publish it.
The advantage of following this system, is that "everybody" agrees your file has not been modified since signing.
The easiest way to achieve this scheme, is to use a resource. Windows resources can be added post- linking. They are protected by the authenticode digital signature, and your program can extract the resource data from itself.
It used to be possible to increase the signature to include binary data. Unfortunately this has been banned. There were binaries which used data in the signature section. Unfortunately this was used maliciously. Some details here msdn blog
Breaking the signature
If re-signing is not an option, then the result would be treated as insecure. It is worth noting here, that appended data is insecure, and can be modified without people being able to tell, but so is the code in your binary.
Appending data to a binary does break the digital signature, and also means the end-user can't tell if the code has been modified.
This means that any self-protection you add to your code to ensure the data blob is still secure, would not prevent your code from being modified to remove the check.
Running module
Windows GetModuleFileName allows the running path to be found.
Linux offers /proc/self or /proc/pid.
Unix does not seem to have a method which is reliable.
Data reading
The approach of the zip format, is to have a directory written to the end of the file. This means the data can be found at the end of the location, and then looked backwards for the start of the data. The advantage here, is the data blob is signposted from the end of the data, rather than the natural start.
So I've run into an interesting design pattern and I wanted to know if you guys had an opinion on it.
Basically, the design is passing everything around as a pre-serialized type. There is no "types" for the returns, for example. It is passed as a simple uint8_t*. There is a defined header that "tells" you what is in the buffer, how big it is, what the version of the buffer is, ect. I call it "pre-serialized" because it forces flattening of all structures.
The pros:
You can easily write it (or even a set of it) to what ever you want. Files, IO, whatever.
Can store arbitrary data.
The Cons: IMHO:
No type safety is going to be a nightmare
The programmer has to parse the code. Even if there is an enumerated type, the user would have to know what that type means. Even if there are functions to parse the type, the programmer has to know that is the function to call.
Version hell: changing code will cause a ripple effect of errors. Because everywhere is parsing it differently, you have no idea where the code works or where it is broken.
It is viral: because it is flat, you can't "insert" the header on the end of outside data. You could wrap the call if you copy your "data", but this could cause an unnecessary copy that would be SLOW. So either your code is slower than it needs to be, or you conform to this data structure.
It isn't human readable OR debug-able.
Have you seen this design pattern before? Is there a name for this design pattern? Things I missed?
Is there a name for this design pattern?
Well, Legacy Code? :) I have seen such design in 30 years old Cobol systems...
The pros you have stated are easily reachable also by using XML format (or JSON):
You can easily write it (or even a set of it) to what ever you want. Files, IO, whatever - most of all, web services!
Can store arbitrary data.
Furthermore, all your cons are eliminated.
The only pro I can see in your solution is conciseness - when every byte counts and you need to avoid any overhead as too expensive, then this is nice.
Added: Cobol has a feature to easily define the structure of such serialized data, see PICTURE clause. Reading the data is very easy then, you read them as variables. (Like if you have a binary data and define a struct in the C language and typecast the binary to the struct.)
As Honza said this would be normal in Legacy Cobol/PL1 (was there a Cobol/PL1 conversion or interface to COBOL programs ???).
In COBOL this design pattern would make sense, not sure about C though (one of the binary serialization packages or JSON etc might be more sensible).
In Cobol, you would have a Cobol copybook which all programs would use and could edit the data using the Cobol Copybook (with something like file-aid or Microfocus Data Editor).
Why use this "design pattern" in Cobol:
Regression testing of Modules; you can write a driver module like
Read Test-data-file
while more-data
Call Module
write Result to output-file
Read Test-data-file
end
You can then do a compare between Output from the
re-Change Program to the changed program.
Testing - some times you can use a "production file" in testing
A file provides trace or snapshot of what is going on, this can be very useful.
Easy to reorganize Batch streams:
Split a programs up (and pass the data via file). There variety of reason for doing this including
program has gotten to big and is hard to maintain.
Sorting the data
Performance (use a file rather than hitting the DB multiple times)
new uses for extracted data
While your cons are valid for C, they will be less of an issue in Cobol.
The key to using this "design pattern" is being able to edit/view/compare the format. If you can not edit/view/compare a file, I do not see the point
I am using C and want to know are XML messages are preferable over text messages as far as communication over a socket connection is concerned?
Is there any other good option available rather to go for XML?
Which is the best parser(or parsing option) available for parsing XML in C?
Is there any standard library which comes with C and helps to parse XML messages?
You design the protocol so you decide. You can use text or binary communication. Whatever format you use, you decide the how to serialize/de-serialize and interpret data. If you use XML, you can leverage on XMLRPC or SOAP. You can use JSONRPC as well. Last time in my project, I used binary in a very simple yet efficient way: The first to identify the method/function to call. The next 2 bytes to inform the length of data (up to 64K - 1 bytes) and the rest is data. Take note of Big/Small Endianess.
It's very subjective. You could use validating or non-validating parsers. TinyXML is lightweight one. You can look into MiniXML and Expat. libxml2 is fatter.
So far XML parsing is not in standard libraries of C or C++. You could use the aforementioned libraries.
Good luck!
EDIT:
By the way, if you want to use binary format to exchange data, just use any of these 3:
http://tpl.sourceforge.net/ - C serialization library.
http://www.s11n.net/c11n/ - A powerful and complicated C serialization library.
Siseria - http://sourceforge.net/projects/siseria/ , purely in C. I wrote for an embedded system project. It runs without any dependency and is very fast! Compared to other 2, mine is very simple and does not use heap and dynamic memory at all. Everything is on the stack!
There are any number of possible solutions. I would look at a few other options before picking XML, I think. XML has quite a lot of overhead; unless you're going to compress your streams it might be a bit costly. XML also isn't easy to edit for humans, although of course more so than a binary format.
You might want to look at JSON, it's a very popular format and is far simpler than XML. There are plenty of implementations available.
I can highly recommend Protobuf, Google's data interchange format. We're using it for communicating between two processes, at it works great. It has built-in support for C++, Python, and Java, and 3-rd party libraries for a bunch of others (Jon Skeet maintains the C# port).
The main question is what are your performance requirements.
If you are going to send one message per second, feel free. If you have human interface on one end, use XML or any other text format.
If you design a machine-to-machine interface, you'd rather consider binary data. Remember to convert everything to network-standard byte order in this case.
Why should I use a human readable file format in preference to a binary one? Is there ever a situation when this isn't the case?
EDIT:
I did have this as an explanation when initially posting the question, but it's not so relevant now:
When answering this question I wanted to refer the asker to a standard SO answer on why using a human readable file format is a good idea. Then I searched for one and couldn't find one. So here's the question
It depends
The right answer is it depends. If you are writing audio/video data for instance, if you crowbar it into a human readable format, it won't be very readable! And word documents are the classic example where people have wished they were human readable, so more flexible, and by moving to XML MS are going that way.
Much more important than binary or text is a standard or not a standard. If you use a standard format, then chances are you and the next guy won't have to write a parser, and that's a win for everyone.
Following this are some opinionated reasons why you might want to choose one over the other, if you have to write your own format (and parser).
Why use human readable?
The next guy. Consider the maintaining developer looking at your code 30 years or six months from now. Yes, he should have the source code. Yes he should have the documents and the comments. But he quite likely won't. And having been that guy, and had to rescue or convert old, extremely, valuable data, I'll thank you for for making it something I can just look at and understand.
Let me read AND WRITE it with my own tools. If I'm an emacs user I can use that. Or Vim, or notepad or ... Even if you've created great tools or libraries, they might not run on my platform, or even run at all any more. Also, I can then create new data with my tools.
The tax isn't that big - storage is free. Nearly always disc space is free. And if it isn't you'll know. Don't worry about a few angle brackets or commas, usually it won't make that much difference. Premature optimisation is the root of all evil. And if you are really worried just use a standard compression tool, and then you have a small human readable format - anyone can run unzip.
The tax isn't that big - computers are quick. It might be a faster to parse binary. Until you need to add an extra column, or data type, or support both legacy and new files. (though this is mitigated with Protocol Buffers)
There are a lot of good formats out there. Even if you don't like XML. Try CSV. Or JSON. Or .properties. Or even XML. Lots of tools exist for parsing these already in lots of languages. And it only takes 5mins to write them again if mysteriously all the source code gets lost.
Diffs become easy. When you check in to version control it is much easier to see what has changed. And view it on the Web. Or your iPhone. Binary, you know something has changed, but you rely on the comments to tell you what.
Merges become easy. You still get questions on the web asking how to append one PDF to another. This doesn't happen with Text.
Easier to repair if corrupted. Try and repair a corrupt text document vs. a corrupt zip archive. Enough said.
Every language (and platform) can read or write it. Of course, binary is the native language for computers, so every language will support binary too. But a lot of the classic little tool scripting languages work a lot better with text data. I can't think of a language that works well with binary and not with text (assembler maybe) but not the other way round. And that means your programs can interact with other programs you haven't even thought of, or that were written 30 years before yours. There are reasons Unix was successful.
Why not, and use binary instead?
You might have a lot of data - terabytes maybe. And then a factor of 2 could really matter. But premature optimization is still the root of all evil. How about use a human one now, and convert later? It won't take much time.
Storage might be free but bandwidth isn't (Jon Skeet in comments). If you are throwing files around the network then size can really make a difference. Even bandwidth to and from disc can be a limiting factor.
Really performance intensive code. Binary can be seriously optimised. There is a reason databases don't normally have their own plain text format.
A binary format might be the standard. So use PNG, MP3 or MPEG. It makes the next guys job easier (for at least the next 10 years).
There are lots of good binary formats out there. Some are global standards for that type of data. Or might be a standard for hardware devices. Some are standard serialization frameworks. A great example is Google Protocol Buffers. Another example: Bencode
Easier to embed binary. Some data already is binary and you need to embed it. This works naturally in binary file formats, but looks ugly and is very inefficient in human readable ones, and usually stops them being human readable.
Deliberate obscurity. Sometimes you don't want it obvious what your data is doing. Encryption is better than accidental security through obscurity, but if you are encrypting you might as well make it binary and be done with it.
Debatable
Easier to parse. People have claimed that both text and binary are easier to parse. Now clearly the easiest to parse is when your language or library supports parsing, and this is true for some binary and some human readable formats, so doesn't really support either. Binary formats can clearly be chosen so they are easy to parse, but so can human readable (think CSV or fixed width) so I think this point is moot. Some binary formats can just be dumped into memory and used as is, so this could be said to be the easiest to parse, especially if numbers (not just strings are involved. However I think most people would argue human readable parsing is easier to debug, as it is easier to see what is going on in the debugger (slightly).
Easier to control. Yes, it is more likely someone will mangle text data in their editor, or will moan when one Unicode format works and another doesn't. With binary data that is less likely. However, people and hardware can still mangle binary data. And you can (and should) specify a text encoding for human-readable data, either flexible or fixed.
At the end of the day, I don't think either can really claim an advantage here.
Anything else
Are you sure you really want a file? Have you considered a database? :-)
Credits
A lot of this answer is merging together stuff other people wrote in other answers (you can see them there). And especially big thanks to Jon Skeet for his comments (both here and offline) for suggesting ways it could be improved.
It entirely depends on the situation.
Benefits of a human readable format:
You can read it in its "native" format
You can write it yourself, e.g. for unit tests - or even for real content, depending on what it's for
Probable benefits of a binary format:
Easier to parse (in terms of code)
Faster to parse
More efficient in terms of space
Easier to control (any time you need text in there, you can ensure it's UTF-8 encoded, and length prefixed etc)
Easier to include opaque binary data efficiently (images, etc - with a text format you'd be getting into base64)
Don't forget that you can always implement a binary format but produce tools to convert to/from a human-readable format as well. That's what the Protocol Buffers framework does - it's actually pretty rare IME to need to parse a text version of a protocol buffer, but it's really handy to be able to write it out as text.
EDIT: Just in case this ends up being an accepted answer, you should also bear in mind the point made by starblue: Human readable forms are much better for diffing. I suspect it would be feasible to design a binary format which is appropriate for diffing (and where a human-readable diff could be generated) but out-of-the-box support from existing diff tools will be better for text.
Version control is easier with text formats, because changes can easily be viewed and merged.
Especially MS-Word is giving us grief in this respect.
Open format -- no binary bit juggling
Readability :)
Interchange across platforms
Debugging aid
Easily parsed (and easily converted to any format)
One important point: you write a parser once, but read the output many times. That kind of tilts the balance in favor of HRF.
A major reason is that if someone needs to read the data say, 30 years from now, human readable format can be figured out. Binary is much more difficult.
If your have large data sets that are binary by nature (e.g. images), they obviously can't be stored in any other than binary form. But even then, the metadata could (and should!) be human-readable.
There's something called The Art of Unix Programming.
I won't say it's good or bad, but it's fairly famous. It has a whole chapter called Textuality in which the author asserts that human readable file format are an important part of the Unix way of programming.
They open the possibility to be created/edited with tools other than the original ones. New and better tools can be developed by others, integration into third party applications becomes possible. Think about binary iCal files, for example - would the format have been a success?
Apart from that: Human readable files improve the ability to debug or, for the savvy user, at least find the reason an error.
Pros for binary:
fast to parse
generally smaller data
easy to write a parser for
Pros for human readable:
easier to understand while reading - no "field X is set to 4 487 which means that the reactor should be shut down NOW"
if using something like XML easy to write a tool that will parse any file
I have had to deal with both types. If you are sending data and you want to keep it small binary is good. If you expect people to read it then human readable is good.
Human readable generally somewhat self documenting as well. And with binary it is bery easy to make mistakes - and hard to spot them.
Editable
Readable (duh!)
Printable
Notepad and vi enabled
Most importantly , their function can be decuded from the content (well mostly)
Because you are a human, and sooner or later you (or one of your customers) will be able to read the data.
We only use binary format if speed is an issue. And even then debugging is troublesome so we added a human readable equivalent.
Interoperability is the standard argument, i.e. a human-readable form is easier for developers of disparate systems to deal with so therefore confers some advantage.
Personally I think that is not true, and the performance benfits of binary files ought to beat that argument, especially if you publish your protocol. However the ubiquity of XML/HTTP based frameworks for machine interactions means that it is easier to adopt.
XML is way over-used.
Just a quick illustration where human-readable document format can be a better choice:
documents used for deploying application in production
We used to have our release notes in word format, but that release notes document had to be opened on various environment (Linux, Solaris) in pre-production and production plateform.
It also had to be parsed in order to extract various data.
In the end, we switched to a wiki-based syntax, still displayed nicely in HTML through a wiki, but still used as a simple text file in other situation.
As an adjuct to this, there are differing levels of human readability, and all are enhanced by using a good editor or viewer with code coloring, folding or navigation.
For example,
JSON is quite readable even in plaintext
XML has the angle bracket tax but is usable when using a good editor
INI is mostly human readable
CSV can be readable, but is best when loaded into a spreadsheet.
No one said, so I will: human-readability is not really a property of a file format (all files are binary after all), but rather of a file format and viewer app combination.
So called human readable formats are all based on top of additional abstraction layer of one of existing text encodings. And viewer programs (often also serving as an editor) that are capable of rendering these encodings in a form readable by humans are very common.
Text encoding standards are widespread and fairly mature, which means they're unlikely to evolve much in the foreseeable future.
Usually on top of the text encoding layer of the format we find a syntax layer that is reasonably intuitive given target user knowledge and cultural background.
Hence the benefits of "human-readable" formats:
Ubiquity of suitable viewers and editors.
Timelessness (given that cultural conventions won't change much).
Easiness-to-learn, read and modify.
Reliance on the extra abstraction layer makes text encoded files:
Space hungry.
Slower to process.
"Binary" files do not resort to text encoding abstraction layer as a base (or a common denominator), but they might or might not use some sort of an extra abstraction more suitable for their purpose and hence, they can be much better optimised for a specific task at hand meaning:
Faster processing.
Smaller footprint.
On the other hand:
Viewers and editors are specific for a particular binary format and make interoperability harder.
Viewers for any given format are less wide spread, because they are more specialised.
Formats might evolve significantly or go out of use over time: their main benefit in being very well suited for a particular task and as the task or task requirements evolve, so does the format.
Take a moment and think about application OTHER than web development.
The assumption that:
A) It has a meaning that is "obvious" in text format is false.
Things like control systems for a steel mill, or manufacturing plant don't typically have any advantage in being human readable. The software for those types of environments will typically have routines to display data in a graphically meaningful manner.
B) Outputting it in text is easier. Unnecessary conversions that actually require more code make a system LESS robust. The fact of the matter if you are NOT using a language which treats all variables as strings then human readable text is an extra conversion. I.E. Extra code means more code to be verified, tested and more opportunities to intro errors in the application.
C) You have to parse it anyway. It many cases for DSP systems I've worked on (I.E. NO Human readable interface to start with.) Data is streamed out of the system in uniformly sized packets. Logging the data for analysis and later processing is simply a matter of pointing to the beginning of a buffer and writing a multiple of the block size to the data logger system. This allows me to analysis the data "untouched" as the customer's system would see it where, once again, converting it to a different format would result in possibly introducing errors. Not only that, if you only save the "converted data" you may lose information in the translation that may help you diagnose a problem.
D) Text is a Natural format for the data. No hardware I've ever seen uses a "TEXT" interface. (My first job out of college was writing a device driver for a camera line scan camera.) The system build on top of it does MIGHT, but for every "PC".
For web pages where the information has a "natural" meaning in text format, so sure knock yourself out. For processing source code it’s a no brainer, of course. But the pervasive computing environments where even you refrigerator and TOOTHBRUSH are going to have a processor built in, not so much. Simply burdening these type of systems with the overhead of adding the ability to process text introduces unnessary complexity. You're not going to link "printf" into the software for an 8-bit micro that controls a mouse. (And yeah, somebody has to write that software too.)
The world is not a black and white place where the only forms of computing that need to be consider are PCs and Web servers.
Even on a PC, if I can directly load the data directly into a datastructure using a single OS read call and be done with it without writing serialize and deserializing routines, that's fantastic, check a blocks CRC job -- done on to the next problem.
Uhm… because human-readable file formats can be read by humans? Seems like a pretty good reason to me.
(Well, for configuration files it’s inevitable that they are read (and edited!) by humans. Files for persistent storage of some sort or the other don’t really need to be read or edited by humans.)
Why should I use a human readable file
format in preference to a binary one?
Is there ever a situation when this
isn't the case?
Yes, compressed volumes (zip, jpeg, mp3, etc) would be suboptimal if they were human readable.
I guess its not good in most situations probably. I think the main reason for these formats such as JSON and XML is because of web development, and general use over the web where you need to be able to process data on the user-side and you cant necessarily read binary. A good example of a bad case to use a human readable format would be any thing non textual such as images, video, audio. Ive noticed the use of non-binary formats being used in web development where it does not make sense, I feel guilty!
Often files become part of your human interface thus they should be human friendly (not programmer only)
The only time that I use a binary stream for files that aren't archives is when I want to conceal things from the casual observer. For instance, if I'm making temporary files that only my application should be editing, I'll use binary.
Its not an attempt to obfuscate, rather its just discouraging the user from editing the file by hand (which could break the application).
One instance where this would be a good idea is storing / saving running data about some game.. i.e. to save your game and continue later. Other scenarios would describe intermediate files, but those are typically binary / byte compiled anyway.
Why should I use a human readable file
format in preference to a binary one?
Depends on the content and context, i.e. where is the data coming from and going. If the data is typically directly written by a human, storing it in an format that can be manipulated through a text editor is a good idea. For example, program source code will normally be stored as human readable with good reason. However, if we are archiving it, or sharing it using a version control system, our storage strategy will change.
The human format is simplier to parsing and debugging if you have a problem with a field (example: a field contains a number where the spec says the this field must be a string), also the human format is closier to domain of problem.
I prefer the binary format with a lot of data AND i'm sure that I have the software for parsing him :)
When reading Fielding's dissertation about REST, I really liked the concept of "Architectural Properties"; one that sticked was "Visibility". That's what we're talking about here: being able to 'see' the data. Huge benefits when debugging the system.
One aspect that I find missing in the other answers: enforcing semantics.
From the moment you go for human readable, you allow the silly notepad user to create data to be fed into the system. No way to guarantee this data makes sense. No way to guarantee the system will respond in a sensible way.
So in the case you don't need to notepad-inspect your data, and you want to enforce valid data (by e.g. usage of an API) rather than first validating it, you better avoid human readable data. If debuggeability is an issue (it most often is), inspection of the data can be done by using the API, too.
Human readable is not equal to easier to be parsed by machine code.
Take human natural language as an example. :) Machine parsing of human language is still a pending problem to be fully solved.
So I agree with https://stackoverflow.com/a/714111/2727173 which has much deeper insight on this question.
I have always been able to read and write basic text files in C++, but so far no one has discussed much more than that.
My question is this:
If developing a file type by myself for use by an application I also create, how would I go about writing the data to a file and preserve the layout, formatting, etc.? Are there any standards, or does it just depend on the creativity of the programmer?
You basically have to come up with your own file format and write binary data.
You can also serialize your object model and write the output to a file, but that's usually less efficient.
Better to use an existing database, or use xml (or other) for simple needs. If you want to write a file in a format that already exists, find a library that supports it.
You have to know the binary file format for the file you are trying to create. Consider Joel's post on this topic: the 97-2003 File Format is a 349 page spec.
Nearly all the time, to do something like that, you use an API, to avoid the grunt work. Be careful however, because trial and error and figuring out "what works" by trial and error can result in an upgrade of the program breaking your code. Plus you have to take into account other operating systems, minor version differences, patches, etc.
There are a number of standards of course. The likely one to use is some flavor of xml since there are libraries and tools that already exist to help you work with it, but nothing is stopping you from inventing your own.
Well you could store the data in a format you could read, but which maintained the integrity of your data (XML or JSON for instance).
Or (shudder) you could come up with your own propriatory binary format, and use that.
you would go at it exactly the same way as you would a text file. writing your data byte by byte, encoded in such a way that when you read the file you know what you are reading.
for a spreadsheet application you could even use a text format (OOXML, OpenDocument) to store presentation and content information.
Or you could define binary datastructures and write that directly to the file.
the choice between text or binary format depends on the application. for a configuration file you may prefer a text file which can be modified outside your app, for a database you will most likely choose a binary format for performance reasons.
See wotsit.org for information on file formats for various file types. Example: You can figure out exactly how to write out a .BMP file and how it is composed.
Writing to a database can be done by using a wrapper class in your language, mainly passing it SQL commands.
If you create a binary file , you can write any file to it . The only drawback is that you have to know exactly where it starts and where it ends .
Use xml (something open, descriptive, and validatable), and stick with the text. There are standards for this sort of thing as well, including ODF
You can open the file as binary, instead of text (how one does this depends somewhat on the platform), from there you can write the data directly out to disk. The only real caveat to this is endianess, which can become an issue when moving the files from one architecture to another (x86 to PPC for instance).
Writing binary data to disk is really no harder than writing text, and really, your creativity is key for how you store the data.
The general problem is usually referred to as serialization of your application state and in your case with a source/target of a file in whatever format makes sense for you. These days the preferred input/output format is XML, and you may want to look into the existing standards in this field. The problem then becomes how do I map from the state of my system to the particular schema. Boost has a serialization framework that you may want to check out.
/Allan
There are a variety of approaches you can take, but in general you'll want some sort of serialization library. BOOST::Serialization, or Google's Protocal Buffers are a good example of these. The basic idea is that you have memory structures (classes and objects) that represent your data, and you want to write that data to a file in a way that can be used to reconstruct those structures again.
If you're hesitant to use a library, you can do it all manually, but realize that you can end up writing a lot of redundant code, or developing your own library. See fopen, fread, fwrite and fclose for a starting point.
A typical binary file format for custom data is an "indexed file format" consisting of
-------
|index|
-------
|data |
-------
Where the index contains records "pointing" to the data.
The index consists of records containing an offset and a size. The offset tells you where in the file the data is stored and the size tells you the size of the data at that offset (i.e. the number of bytes to read).
typedef struct {
size_t offset
size_t size
} Index
typedef struct {
int ID
char First[20]
char Last[20]
char *RandomInfo
} Data
Suppose you want to store 50 records in the file you would create 50 indices and 50 data structures. The 50 index structures would be written to the file first, followed by the 50 data structures.
To read the file you would read in the 50 index structures, then from the data in the read-in index structures you could tell where to "seek" to read the data records.
Look up (fopen, fread, fwrite, fclose, ftell) for functions to read/write the data.
(Sorry my semicolon key doesn't work)
You usually use a third party library for these things. For example, you would link in a database library for say Oracle that would allow you to talk to the database. Because the underlying file type, ( i.e. Excel spreadsheet vs Openoffice, Oracle vs MySQL, etc. ) differ these libraries abstract away your need to care how the file is constructed.
Hope that helps you find what you're looking for!
1985 called, and said they have some help IFF you are willing to read up. The interchange file format is still in use today and provides some basic metadata around binary files, such as RIFF or WAV audio. (Unfortunately, TIFF is a false friend.) It allegedly even inspired PNG, so it can't be that bad.