So, I’m working on a plain-C (ANSI 9899:1999) project, and am trying to figure out where to get started re: Unicode, UTF-8, and all that jazz.
Specifically, it’s a language interpreter project, and I have two primary places where I’ll need to handle Unicode: reading in source files (the language ostensibly supports Unicode identifiers and such), and in ‘string’ objects.
I’m familiar with all the obvious basics about Unicode, UTF-7/8/16/32 & UCS-2/4, so on and so forth… I’m mostly looking for useful, C-specific (that is, please no C++ or C#, which is all that’s been documented here on SO previously) resources as to my ‘next steps’ to implement Unicode-friendly stuff… in C.
Any links, manpages, Wikipedia articles, example code, is all extremely welcome. I’ll also try to maintain a list of such resources here in the original question, for anybody who happens across it later.
A must read before considering anything else, if you’re unfamiliar with Unicode, and what an encoding actually is: http://www.joelonsoftware.com/articles/Unicode.html
The UTF-8 home-page: http://www.utf-8.com/
man 3 iconv (as well as iconv_open and iconvctl)
International Components for Unicode (via Geoff Reedy)
libbasekit, which seems to include light Unicode-handling tools
Glib has some Unicode functions
A basic UTF-8 detector function, by Christoph
International Components for Unicode provides a portable C library for handling unicode. Here's their elevator pitch for ICU4C:
The C and C++ languages and many operating system environments do not provide full support for Unicode and standards-compliant text handling services. Even though some platforms do provide good Unicode text handling services, portable application code can not make use of them. The ICU4C libraries fills in this gap. ICU4C provides an open, flexible, portable foundation for applications to use for their software globalization requirements. ICU4C closely tracks industry standards, including Unicode and CLDR (Common Locale Data Repository).
GLib has some Unicode functions and is a pretty lightweight library. It's not near the same level of functionality that ICU provides, but it might be good enough for some applications. The other features of GLib are good to have for portable C programs too.
GTK+ is built on top of GLib. GLib provides the fundamental algorithmic language constructs commonly duplicated in applications. This library has features such as (this list is not a comprehensive list):
Object and type system
Main loop
Dynamic loading of modules (i.e. plug-ins)
Thread support
Timer support
Memory allocator
Threaded Queues (synchronous and asynchronous)
Lists (singly linked, doubly linked, double ended)
Hash tables
Arrays
Trees (N-ary and binary balanced)
String utilities and charset handling
Lexical scanner and XML parser
Base64 (encoding & decoding)
I think one of the interesting questions is - what should your canonical internal format for strings be? The 2 obvious choices (to me at least) are
a) utf8 in vanilla c-strings
b) utf16 in unsigned short arrays
In previous projects I have always chosen utf-8. Why ; because its the path of least resistance in the C world. Everything you are interfacing with (stdio, string.h etc) will work fine.
Next comes - what file format. The problem here is that its visible to your users (unless you provide the only editor for your language). Here I guess you have to take what they give you and try to guess by peeking (byte order marks help)
Related
Are there any widely used I/O stream abstraction layers for plain C?
By an I/O stream abstraction layer I mean any layer that at least allows the creation of custom read/write functions. For C++, there's the standard iostream and boost::iostreams. For glibc users, there's a possibility to use custom streams. These won't do any good if the requirement is to write portable C code.
It is better to use a library that is either widely used or easy to embed to the source code, or both.
SDL_RWops is an undocumented but widely used feature of SDL, and zziplib can use it. However, it does not make much sense to add a dependency to SDL just to get this feature.
GLib contains a GIOChannel abstraction, but the library is again quite large and they say that "support for Windows is only partially complete".
However, the above are not quite satisfactory for small libraries, such as decoders, file format readers and signal processors: they contain lots of unnecessary stuff and the LGPL licensing prevents embedding the relevant parts to non-GPL code.
The BIO abstraction in OpenSSL sounds like it fits the bill.
I think you answered your own question.
No, there are no widely used steam abstraction libraries. Those that exist are usually a small part of libraries that most people don't want to depend on, or are very specialized.
What are your requirements? What is it you are looking for out of an abstraction library? I have to admit that every time I've looked at one, or started writing one, I usually end up back at the standard POSIX interfaces... what more abstraction could one want?
There is libslack (GNU GPL) which may provide some of the functionality you are after and this MIT License-ed input stream wrapper:
http://attractivechaos.wordpress.com/2008/10/11/a-generic-buffered-stream-wrapper/
The lexical analyzer generator Quex comes along with an input stream abtraction for POSIX, C++ stream, etc. It consists of a collection of classes derived from a class called ByteLoader.
It is easily adaptable to any infrastructure-simply by deriving from ByteLoader and implemting the interface. See: Link to Code.
A 'real' ByteLoader is only instantiated upon generation of a lexer, though.
I'm searching for something on the level of GNU extensions for C, but a little beyond (some basic data structure management). Best would be something BSD/MIT licensed.
If there is something for just strings containing GNU extensions equivalents plus adding some more it would be great.
I would prefer something that can be simply compiled into a project (no external libraries) based totally on the C standard (ANSI C89 if possible).
Edit: its for an OpenSource project that has a weird license, so no GPL code can't be added and working with plain K&R/ANSI C is pure pain.
This question already seems to be addressed here.
I actually wrote a somewhat lengthy response (recommending Glib and mentioning that Lua since 5.0 is MIT, not BSD), however my machine crashed half-way through :(
Thinking outside the box, a viable but off-the-wall approach is to use Lua. It is small, written in the subset of ANSI C that also happens to be valid C++, and supplies a rich garbage collected environment for strings, and associative arrays.
It can be built as a shared library, but it can also be statically linked.
It can admittedly feel a tad verbose when driving its data types entirely from the C side, but it is easy to move some of the higher level logic of your application into the Lua side where its data just works. Its VM is highly tuned, allowing it to perform better than one would expect for an interpreted scripting language, and there is a JIT compiler available as well for those times when its existing VM just isn't quite fast enough.
It is also open source and MIT licensed.
Everyone knows how awesome C language is and how much it sucks in text processing tasks. Given these facts. Regex definitely must be part of ISO C. But it isn't. I don't understand why? Are there people who think its not essential?
Regular Expressions don't belong in the C language proper any more than a sound library, a graphics library, or an encryption library does. Doing so would reduce the general purpose nature of the language and greatly inhibit its use as a small and efficient embedded language.
The philosophy of C was to have a very small and efficient language keyword set with standardized libraries for the next layer of functionality. Since things like regex, graphics, sound, encryption, etc. don't have a single platform or standard they don't fit in with the standard C library.
They fit best as user libraries which they currently are.
Regex is defined as part of IEEE Std 1003.1:2001 (POSIX)
Here's a handly list of which headers are in which standard:
http://www.schweikhardt.net/identifiers.html
Because it is a library feature that would require standardizing on one of the regex languages. Standard bodies are commitee driven, not an easy task.
This document explains the rationalization of the standard: http://www.open-std.org/jtc1/sc22/wg14/www/docs/C99RationaleV5.10.pdf which might clarify why.
Another reason explained in the doc. is to keep the language simple.
There are quite a few downloads available, just use one.
Because regexes are not essential to a programming language. Handy? Yes, very much so, when you need them. Essential? No way.
Web developers will naturally consider regexes to be an essential feature of a language because they have to validate all that HTML form data. Developers whose experience is always with one of a few big-name relational database servers will consider SQL support to be essential. Those working in the scientific domain will require support for "big numbers" or tensors. GUI developers think a built-in GUI toolkit is essential. Some folks deal with XML all day and consider XML support to be essential.... etc. you get the idea. This list of "essentials" can get pretty big, and languages like Java have certainly taken the "kitchen sink" approach to their massive standard libraries. I appreciate that C is not a kitchen sink language in that sense.
Be careful not to assume that your favorite language feature is an essential feature for everyone else.
The point of C is to be small yet powerful. Since regular expressions are typically a large and complex topic, it belongs in a library. It is too bad though that the C committee doesn't "sponser" some well written, standard C, algorithms/data structure libraries. There is a plethora of them out there. I tend to stick with GNU "sponsored" libs whenever I can since they are available for most platforms even if they aren't necessarily the easiest or most efficient to use. They do strike a nice balance.
I've been working on a C program which does quite a lot of string manipulation, and very often needs to be tweaked and recompiled for some sort of special case processing. I've been thinking that embedding some scripting language with good string manipulation support might make sense for the project.
What language would provide the best string manipulation support while being easy to embed in a C program?
For some extra background...
Performance is pretty important (especially startup time)
Needs easily be compiled on multiple platforms (Linux, Solaris, Win32 (ideally with MinGW), Darwin)
Needs to be a language which will still be around in 5 years time
I've looked a little at Python (perhaps too heavy weight?) and Lua (perhaps not focused on string manipulation?) but don't really know enough about them or what other choices might be out there.
I've never regretted using Lua.
It's very easy to embed in your application. In fact, now I usually don't write C applications, i just write C libraries and control them from Lua.
Text manipulation isn't its best feature, but it's certainly far better than C alone. And the LPEG library makes building parsers almost trivially easy, putting any regex to shame (but still has a couple of regex-like syntaxes if you prefer them).
Lua stands head and shoulders above other choices.
... best string manipulation support while being easy to embed?
Lua is designed to be embedded in C; the API is clear and easy to use; the documentation is terrific.
Some other responses have denigrated Lua's string capabilities. I think they're underestimating Lua.
Lua's string capabilities actually find a sweet spot between "just concatenation" and the full complexity of regular expressions. String formatting capability is very strong, and accumulating strings through "buffers" or tables is simple and efficient.
String scanning is, in my opinion, one of the best parts of the design. It doesn't have "or" patterns but otherwise gives you a large fraction of what you get from regular expressions, including a very powerful and elegant "capture" function. For example, I can convert a string to hex by capturing every single character and applying a function to it:
s:gsub('.', function(c) return string.format("%02x", string.byte(c)) end)
Or I can escape non-alphanumeric, non-space characters into octal:
s:gsub('[^%w%s]', function(c) return string.format([[\%03o]], string.byte(c)) end)
Some of the features on display here:
The escape character for string scanning is %, which is different from the escape character for string quoting, which is \. This decision is brilliant and should win an award by itself :-)
There are multiple mechanisms for quoting literal strings, including [[...]] in which no characters have to be escaped. If you want to generate or match strings with backslashes in them (like LaTeX for example), this is a godsend.
If you want the full power of a context-free parser, you can always use LPEG, a library written by one of Lua's designers.
Performance is pretty important (especially startup time)
Lua consistently wins performance awards. Startup is lightning fast: the whole system (including compiler, library, garbage collector, and runtime system) fits in 150KB. To avoid pause times, Lua provides incremental garbage collection. See also SO question Why is Lua faster than other scripting languages?
You can make startup even faster by precompiling your scripts, but I've never found it necessary to do this—and because compiled code (as opposed to source code) is not portable, precompilation usually creates more headache than it solves.
Needs easily be compiled on multiple platforms
Lua compiles using pure ANSI C and does not even require POSIX. I have a version running on my PalmOS PDA.
Needs to be a language which will still be around in 5 years time.
Lua has been around since 1993. Moreover, the two members of the team who provide the most support are tenured professors at PUC-Rio. Lua is their livelihood. Finally, the whole system is only 17,000 lines of code. If Rio fell off the map tomorrow, anybody with a good undergraduate compiler course could pick the system up and maintain it. There would be plenty of volunteers.
I've looked a little at Python and Lua but don't really know enough about them
See SO question Which game scripting language is better to use: Lua or Python?.
People have been embedding tcl in larger projects for what seems like ages. It's been a while since I've had to use tcl for anything...
One of the things that sets tcl apart from other programming languages is that everything is a string.
And for your reference, here's the tcl documentation on string functions.
tcl might be easier to embed than perl, but I do have to agree #Matthew Scharley's reasoning. Also, tcl isn't exactly known for it's performance, but maybe that's changed in recent years.
Anyway, here is the tcl wiki link on embedding tcl in C applications, and a relevant quote from the page:
"How do I embed a Tcl interpreter in my existing C (or C++) application?" is a very frequently-asked question. It's straightforward, certainly far easier than doing the same with Perl or, in general, Python; moreover, this sort of "embeddability" was one of the original goals for Tcl, and many, many projects do it. There are no complete discussions of the topic available, but we can give an overview here. (RWT 14-Oct-2002)
Another alternative might be to go with Lua, as you mentioned, while extending it with another C string library of your choice (Google turns up The Better String Library, for instance).
Once you've compiled Lua into your application, you can "extend" C functions to Lua's interpreter. Or maybe the built-in string functions are adequate for you.
You certainly have a few options.
We looked at both Python and Lua for scripting for a .NET product. The goal was to provide some scripability for end users. The decision came down to Python because the powers-that-be preferred anything with Microsoft support to everything else. My choice was for Lua.
There's a good survey paper on the relative merits of the embedding APIs of various scripting languages:
H. Muhammad and R. Ierusalimschy. C APIs in extension
and extensible languages. Journal of Universal Computer
Science, 13(6):839–853, 2007.
Looking at combining both excellent string manipulation and an excellent embed API, I would suggest, in order:
Ruby: Excellent string support, including syntax support for regex. Well-designed embed API, very easy to use.
Lua: I'm not sure how its string support is, but its supposed to be a great language for embedding.
Python: Less easy to embed, slightly harder to use string features than Ruby. But it has Pyrex, so that might be an easier way to embed it.
PHP: Nasty API, nasty language. The embed SAPI is really a second-class citizen, but it does work. There are a lot of string manipulation functions. Still, I wouldn't recommend it.
Perl: Nasty to embed (so far as I've heard), string support could be better.
I can't comment about TCL, but I hear its designed for embedding.
Python is not heavyweight at all! It's quite simple to embed (here's the official guide, but you can find many tutorials as well), very powerful, great for string processing, and a pleasant and easy language to use overall. It has a huge user community and support base, which is a bonus.
Python has also been embedded into a large number of real-life applications. One cool example I can think of immediately is the Civilization IV game, most of which runs on Python scripts on top of a C++ API.
Some people may disagree but Sara Goleman has published a great book on extending and embedding PHP. Which is becoming one of the most widely used languages around... :)
PHP String support isn't as great as say Perl, but it's very usable.
Did I mention it's written in C?
</my2cents>
Perl. Its (original) reason for being is string manipulation.
I don't ever recall coming across anything for C that's as extensive as C++'s Boost library.
You can have a look at glib. It provides quite a few interesting things like containers, unicode support, threading...
Have a look at its documentation
I've never seen anything. Really, C++ templates are the enabling technology for STL and much of boost, and C doesn't have them. I've seen many C-interface libraries for things like image processing, 3D graphics, networking, etc., but never a real general-purpose library like boost. You can find pieces though - threading libraries, string libraries, etc. - that are similar to sections of boost.
Depending on what exactly you're trying to do there is APR (The Apache Portable Runtime library) which is what the Apache http daemon is built on
http://apr.apache.org/
For GUI, there is wxWidgets (formerly wxWindows)
http://www.wxwidgets.org/
Glib does compensate it partially in form of various unicode, string types. If you add Gobject you could get some object programming even some garbage collection.
ACE (Adaptive Communication Environment) is sometimes mentioned. It's not quite an apples-to-apples comparison. Boost provides more "basic building blocks" whereas ACE provides more of an tightly integrated framework geared towards telco products. I have used both extensively and I find Boost vastly superior. One big advantage of Boost is that many of its features will show up in the STL extensions for the upcoming C++0x standard (see, for example, http://en.wikipedia.org/wiki/Technical_Report_1).
or you can look at ccan http://ccan.ozlabs.org/
but no, there really isn't any complete package, glib comes closest