I'm trying to print out a wchar_t* string.
Code goes below:
#include <stdio.h>
#include <string.h>
#include <wchar.h>
char *ascii_ = "中日友好"; //line-1
wchar_t *wchar_ = L"中日友好"; //line-2
int main()
{
printf("ascii_: %s\n", ascii_); //line-3
wprintf(L"wchar_: %s\n", wchar_); //line-4
return 0;
}
//Output
ascii_: 中日友好
Question:
Apparently I should not assign CJK characters to char* pointer in line-1, but I just did it, and the output of line-3 is correct, So why? How could printf() in line-3 give me the non-ascii characters? Does it know the encoding somehow?
I assume the code in line-2 and line-4 are correct, but why I didn't get any output of line-4?
First of all, it's usually not a good idea to use non-ascii characters in source code. What's probably happening is that the chinese characters are being encoded as UTF-8 which works with ascii.
Now, as for why the wprintf() isn't working. This has to do with stream orientation. Each stream can only be set to either normal or wide. Once set, it cannot be changed. It is set the first time it is used. (which is ascii due to the printf). After that the wprintf will not work due the incorrect orientation.
In other words, once you use printf() you need to keep on using printf(). Similarly, if you start with wprintf(), you need to keep using wprintf().
You cannot intermix printf() and wprintf(). (except on Windows)
EDIT:
To answer the question about why the wprintf line doesn't work even by itself. It's probably because the code is being compiled so that the UTF-8 format of 中日友好 is stored into wchar_. However, wchar_t needs 4-byte unicode encoding. (2-bytes in Windows)
So there's two options that I can think of:
Don't bother with wchar_t, and just stick with multi-byte chars. This is the easy way, but may break if the user's system is not set to the Chinese locale.
Use wchar_t, but you will need to encode the Chinese characters using unicode escape sequences. This will obviously make it unreadable in the source code, but it will work on any machine that can print Chinese character fonts regardless of the locale.
Line 1 is not ascii, it's whatever multibyte encoding is used by your compiler at compile-time. On modern systems that's probably UTF-8. printf does not know the encoding. It's just sending bytes to stdout, and as long as the encodings match, everything is fine.
One problem you should be aware of is that lines 3 and 4 together invoke undefined behavior. You cannot mix character-based and wide-character io on the same FILE (stdout). After the first operation, the FILE has an "orientation" (either byte or wide), and after that any attempt to perform operations of the opposite orientation results in UB.
You are omitting one step and therefore think the wrong way.
You have a C file on disk, containing bytes. You have a "ASCII" string and a wide string.
The ASCII string takes the bytes exactly like they are in line 1 and outputs them.
This works as long as the encoding of the user's side is the same as the one on the programmer's side.
The wide string first decodes the given bytes into unicode codepoints and stored in the program- maybe this goes wrong on your side. On output they are encoded again according to the encoding on the user's side. This ensures that these characters are emitted as they are intended to, not as they are entered.
Either your compiler assumes the wrong encoding, or your output terminal is set up the wrong way.
Related
I'm using libxml/xmlwriter to generate an XML file within a program.
const char *s = someCharactersFromSomewhere();
xmlTextWriterWriteAttribute (writer, _xml ("value"), _xml (s));
In general I don't have much control over the contents of s, so I can't guarantee that it will be well-formatted in UTF-8. Mostly it is, but if not, the XML which is generated will be malformed.
What I'd like to find is a way to convert s to valid UTF-8, with any invalid character sequences in s replaced with escapes or removed.
Alternatively, if there is an alternative to xmlTextWriterWriteAttribute, or some option I can pass in when initializing the XML writer, such that it guarantees that it will always write valid UTF-8, that would be even better.
One more thing to mention is that the solution must work with both Linux and OSX. Ideally writing as little of my own code as possible! :P
In case the string is encoded in ASCII, then it will always be valid UTF-8 string.
This is because UTF-8 is backwards compatible with ASCII encoding.
See the second paragraph on Wikipedia here.
Windows primarily works with UTF-16, this means you will have to convert from UTF-16 to UTF-8 before you pass the string to the XML library.
If you have 8-bit ascii input then you can simply junk any character code > 127.
If you have some dodgy UTF-8 it is quite easy to parse, but the widechar symbol number that you generate might be out of the unicode range. You can use mbrlen() to individually validate each character.
I am describing this using unsigned chars. If you must use signed chars, then >128 means <0.
At its simplest:
Until the null byte
1 If the next byte is 0, then end the loop
2 If the next byte is < 128 then it is ascii, so keep it
3 If the next byte is >=128 < 128+64 it is invalid - discard it
4 If the next byte is >= 128+64 then it is probably a proper UTF-8 lead byte
call size_t mbrlen(const char *s, size_t n, mbstate_t *ps);
to see how many bytes to keep
if mbrlen says the code is bad (either the lead byte or the trail bytes),
skip 1 byte. Rule 3 will skip the rest.
Even simpler logic just calls mbrlen repeatedly, as it can accept the low ascii range.
You can assume that all the "furniture" of the file (eg xml <>/ symbols, spaces, quotes and newlines) won't be altered by this edit, as they are all valid 7-bit ascii codes.
char is a single byte character, while UTF codepoints range from 0 to 0x10FFFFF, so how do you represent a UTF character in only one byte?
First of all you need a wchar_t character. Those are used with wprintf(3) versions of the normal printf(3) routines. If you dig a little on this, you'll see that mapping your UTF codepoints into valid UTF-8 encoding is straigtforward, based on your setlocale(3) settings. Look at those manual pages referenced, and you'll get an idea of the task you are facing.
There's full support for wide character sets in the C standard... but you have to use it through the internationalization libraries and locales availables.
wprintf() takes a wchar_t string as argument and prints the string in the specified locale character encoding.
But I have noticed that when using printf() and passing it a UTF-8 string, the UTF-8 string will always be printed regardless of the specified locale character encoding (for example, if the UTF-8 string contains Arabic characters, and the locale is set to "C" (not "C.UTF-8"), then the Arabic characters will still be printed).
Am I correct that printf() doesn't care about the locale?
True printf doesn't care about locale for c-strings. If you pass it an UTF-8 string, it knows nothing about it, it just see a sequence of bytes (hopefully terminated by ascii NUL). Then, bytes are passed to the output as-is, and are interpreted by the terminal (or whatever is the output). If the terminal is able to interpret UTF-8 sequences it then does so (if not, it tries to interpret it the way it is configured, Latin-1 or alike) and if it is also able to print them correctly then it does so (sometimes it doesn't have the right font/glyph and prints unknown characters as ? or alike).
This is one of the big virtues (perhaps the biggest virtue) of UTF-8: it's just a string of reasonably ordinary bytes. If your code-editing environment knows how to let you type
printf("Cööl!\n");
and if your display environment (e.g. your terminal window) knows how to display it, you can just write that, and run it, and it works (as it sounds like you've discovered).
So you don't need special run-time support, you don't need special header files or libraries or anything, you don't need to write your code in some fancy new Unicodey way -- you can just keep on using ordinary C strings and printf and friends like you're used to, and it all just works.
Of course, those two if's can be big ones. If you can't figure out how to (or your code editing environment won't let you) type the characters, or if your display environment doesn't display them, you may be stuck, or you may have to do some hard work after all. (Display environments that don't properly display UTF-8 output from C programs are evidently quite common, based on the number of times the question gets asked here on SO.)
See also the "UTF-8 Everywhere" manifesto.
(Now, with all of this said, this doesn't mean that printf doesn't care about locale settings at all. There are aspects of the locale that printf may care about, and there may be character sets and encodings that printf might have to treat specially, in a locale-dependent way. But since printf doesn't have to do anything special to make UTF-8 work right, that one aspect of the locale -- although it's a biggie -- doesn't end up affecting printf at all.)
Let's consider the following simple program, which uses printf() to print a wide string if run without command-line arguments, and wprintf() otherwise:
#include <stdlib.h>
#include <locale.h>
#include <stdio.h>
#include <wchar.h>
const wchar_t hello1[] = L"تحية طيبة";
const wchar_t hello2[] = L"Tervehdys";
int main(int argc, char *argv[])
{
if (!setlocale(LC_ALL, ""))
fprintf(stderr, "Warning: Current locale is not supported by the C library.\n");
if (argc <= 1) {
printf("printf 1: %ls\n", hello1);
printf("printf 2: %ls\n", hello2);
} else {
wprintf(L"wprintf: %ls\n", hello1);
wprintf(L"wprintf: %ls\n", hello2);
}
return EXIT_SUCCESS;
}
Using the GNU C library and any UTF-8 locale:
$ ./example
printf 1: تحية طيبة
printf 2: Tervehdys
$ ./example wide
wprintf: تحية طيبة
wprintf: Tervehdys
i.e. both produce the exact same output. However, if we run the example in the C/POSIX locale (that only supports ASCII), we get
$ LANG=C LC_ALL=C ./example
printf 1: printf 2: Tervehdys
i.e., the first printf() stopped at the first non-ASCII character (and that's why the second printf() printed on the same line);
$ LANG=C LC_ALL=C ./example wide
wprintf: ???? ????
wprintf: Tervehdys
i.e. wprintf() replaces wide characters that cannot be represented in the charset used by the current locale with a ?.
So, if we consider the GNU C library (which exhibits this behaviour), then we must say yes, printf cares about the locale, although it actually mostly cares about the character set used by the locale, and not the locale per se:
printf() will stop when trying to print wide strings that cannot be represented by the current character set (as defined by the locale). wprintf() will output question marks for those characters instead.
libc6-2.23-0ubuntu10 on x86-64 (amd64) does some replacements for multibyte characters in the printf format string, but multibyte characters in strings printed with %s are printed as-is. Which means it is a bit complicated to say exactly what gets printed and when the printf() gives up on the first multibyte or wide character it cannot convert, or just prints as-is.
However, wprintf() is pretty rock solid. (It too may choke if you try to print narrow strings with multibyte characters not representable in the character set used by the current locale, but for wide string stuff, it seems to work very well.)
Do note that POSIX.1 C libraries also provide iconv_open(), iconv(), and iconv_close() for converting strings, as well as mbstowcs() and wcstombs() to convert between wide and narrow/multibyte strings. You can also use asprintf() to create a dynamically allocated narrow string out of narrow and/or wide character strings (%s and %ls, respectively).
Ok, I have this:
AllocConsole();
SetConsoleOutputCP(CP_UTF8);
HANDLE consoleHandle = GetStdHandle(STD_OUTPUT_HANDLE);
WriteConsoleA(consoleHandle, "aΕλληνικά\n", 10, NULL, NULL);
WriteConsoleW(consoleHandle, L"wΕλληνικά\n", 10, NULL, NULL);
printf("aΕλληνικά\n");
wprintf(L"wΕλληνικά\n");
Now, the issue is that depending on the encoding file was saved as only some these works. wprintf never works, but I already know why (broken Microsoft stdout implementation, which only accepts narrow characters). Yet, I have issue with three others. If I save file as UTF-8 without signature (BOM) and use MS Visual C++ compiler, only last printf works. If I want ANSI version working I need to increase character(?) count to 18:
WriteConsoleA(consoleHandle, "aΕλληνικά\n", 18, NULL, NULL);
WriteConsoleW does not work, I assume, because the string is saved as UTF-8 byte sequence even I explicitly request it to be stored as wide-char (UTF-16) with L prefix and implementation most probably expects UTF-16 encoded string not UTF-8.
If I save it in UTF-8 with BOM (as it should be), then WriteConsoleW starts to work somehow (???) and everything else stops (I get ? instead of a character). I need to decrease character count in WriteConsoleA back to 10 to keep formatting the same (otherwise i get 8 additional rectangles). Basically, WTF?
Now, let's go to UTF-16 (Unicode - Codepage 1200). Works only WriteConsoleW. Character count in WriteConsoleA should be 10 to keep formatting precise.
Saving in UTF-16 Big Endian mode (Unicode - Codepage 1201) does not change anything. Again, WTF? Shouldn't byte order inside the strings be inverted when stored to file?
Conclusion is that the way strings are compiled into binary form depends on the encoding used. Therefore, what is the portable and compiler independent way to store strings? Is there a preprocessor which would convert one string representation into another before compilation, so I could store file in UTF-8 and only preprocess strings which I need to have in UTF-16 by wrapping them some macro.
I think you've got at least a few assumptions here which are either wrong or not 100% correct as far as I know:
Now, the issue is that depending on the encoding file was saved as only some these works.
Of course, because the encoding determines how to Interpret the string literals.
wprintf never works, but I already know why (broken Microsoft stdout implementation, which only accepts narrow characters).
I've never heard of that one, but I'm rather sure this depends on the locale set for your program. I've got a few work Projects where a locale is set and the output is just fine using German umlauts etc.
If I save file as UTF-8 without signature (BOM) and use MS Visual C++ compiler, only last printf works. If I want ANSI version working I need to increase character(?) count to 18:
That's because the ANSI version wants an ANSI string, while you're passing a UTF-8 encoded string (based on the file's encoding). The output still works, because the console handles the UTF-8 conversion for you - you're essentially printing raw UTF-8 here.
WriteConsoleW does not work, I assume, because the string is saved as UTF-8 byte sequence even I explicitly request it to be stored as wide-char (UTF-16) with L prefix and implementation most probably expects UTF-16 encoded string not UTF-8.
I don't think so (although I'm not sure why it isn't working either). Have you tried Setting some easy to find string and look for it in the resulting binary? I'm rather sure it's indeed encoded using UTF-16. I assume due to the missing BOM the compiler might interpret the whole thing as a narrow string and therefore converts the UTF-8 stuff wrong.
If I save it in UTF-8 with BOM (as it should be), then WriteConsoleW starts to work somehow (???) and everything else stops (I get ? instead of a character). I need to decrease character count in WriteConsoleA back to 10 to keep formatting the same (otherwise i get 8 additional rectangles). Basically, WTF?
This is exactly what I described above. Now the wide string is encoded properly, because the Compiler now knows the file is in UTF-8, not ANSI (or some codepage). The narrow string is properly converted to the locale being used as well.
Overall, there's no encoding independant way to do it, unless you escape everything using the proper codepage and/or UTF codes in advance. I'd just stick to UTF-8 with BOM, because I think all current compilers will be able to properly read and Interpret the file (besides Microsoft's Resource Compiler; although I haven't tried feeding the 2012 Version with UTF-8).
Edit:
To use an analogy:
You're essentially saving a raw image to a file and you expect it to work properly, no matter whether other programs try to read it as a grayscale, palettized, or full color image. This won't work (despite differences being smaller).
The answer is here.
Quoting:
It is impossible for the compiler to intermix UTF-8 and UTF-16
strings into the compiled output! So you have to decide for one source
code file:
either use UTF-8 with BOM and generate UTF-16 strings only (i.e.always use L prefix),
or UTF-8 without BOM and generate UTF-8 strings only (i.e. never use L prefix),
7-bit ASCII characters are not involved and can be used with or without L prefix
The only portable and compiler independent way is to use ASCII charset and escape sequences, because there are no guarantees that any compiler would accept UTF-8 encoded file and a compiler treatment of those multibyte sequences might vary.
I'm primarily interested in the Unix-like systems (e.g., portable POSIX) as it seems like Windows does strange things for wide characters.
Do the read and write wide character functions (like getwchar() and putwchar()) always "do the right thing", for example read from utf-8 and write to utf-8 when that is the set locale, or do I have to manually call wcrtomb() and print the string using e.g. fputs()? On my system (openSUSE 12.3) where $LANG is set to en_GB.UTF-8 they do seem to do the right thing (inspecting the output I see what looks like UTF-8 even though strings were stored using wchar_t and written using the wide character functions).
However I am unsure if this is guaranteed. For example cprogramming.com states that:
[wide characters] should not be used for output, since spurious zero
bytes and other low-ASCII characters with common meanings (such as '/'
and '\n') will likely be sprinkled throughout the data.
Which seems to indicate that outputting wide characters (presumably using the wide character output functions) can wreak havoc.
Since the C standard does not seem to mention coding at all I really have no idea who/when/how coding is applied when using wchar_t. So my question is basically if reading, writing and using wide characters exclusively is a proper thing to do when my application has no need to know about the encoding used. I only need string lengths and console widths (wcswidth()), so to me using wchar_t everywhere when dealing with text seems ideal.
The relevant text governing the behavior of the wide character stdio functions and their relationship to locale is from POSIX XSH 2.5.2 Stream Orientation and Encoding Rules:
http://pubs.opengroup.org/onlinepubs/9699919799/functions/V2_chap02.html#tag_15_05_02
Basically, the wide character stdio functions always write in the encoding that's in effect (per the LC_CTYPE locale category) at the time the FILE stream becomes wide-oriented; this means the first time a wide stdio function is called on it, or fwide is used to set the orientation to wide. So as long as a proper LC_CTYPE locale is in effect matching the desired "system" encoding (e.g. UTF-8) when you start working with the stream, everything should be fine.
However, one important consideration you should not overlook is that you must not mix byte and wide oriented operations on the same FILE stream. Failure to observe this rule is not a reportable error; it simply results in undefined behavior. As a good deal of library code assumes stderr is byte oriented (and some even makes the same assumption about stdout), I would strongly discourage ever using wide-oriented functions on the standard streams. If you do, you need to be very careful about which library functions you use.
Really, I can't think of any reason at all to use wide-oriented functions. fprintf is perfectly capable of sending wide-character strings to byte-oriented FILE streams using the %ls specifier.
So long as the locale is set correctly, there shouldn't be any issues processing UTF-8 files on a system using UTF-8, using the wide character functions. They'll be able to interpret things correctly, i.e. they'll treat a character as 1-4 bytes as necessary (in both input and output). You can test it out by something like this:
#include <stdio.h>
#include <locale.h>
#include <wchar.h>
int main()
{
setlocale(LC_CTYPE, "en_GB.UTF-8");
// setlocale(LC_CTYPE, ""); // to use environment variable instead
wchar_t *txt = L"£Δᗩ";
wprintf(L"The string %ls has %d characters\n", txt, wcslen(txt));
}
$ gcc -o loc loc.c && ./loc
The string £Δᗩ has 3 characters
If you use the standard functions (in particular character functions) on multibyte strings carelessly, things will start to break, e.g. the equivalent:
char *txt = "£Δᗩ";
printf("The string %s has %zu characters\n", txt, strlen(txt));
$ gcc -o nloc nloc.c && ./nloc
The string £Δᗩ has 7 characters
The string still prints correctly here because it's essentially just a stream of bytes, and as the system is expecting UTF-8 sequences, they're translated perfectly. Of course strlen is reporting the number of bytes in the string, 7 (plus the \0), with no understanding that a character and a byte aren't equivalent.
In this respect, because of the compatibility between ASCII and UTF-8, you can often get away with treating UTF-8 files as simply multibyte C strings, as long as you're careful.
There's a degree of flexibility as well. It's possible to convert a standard C string (as a multibyte string) to a wide character string easily:
char *stdtxt = "ASCII and UTF-8 €£¢";
wchar_t buf[100];
mbstowcs(buf, stdtxt, 20);
wprintf(L"%ls has %zu wide characters\n", buf, wcslen(buf));
Output:
ASCII and UTF-8 €£¢ has 19 wide characters
Once you've used a wide character function on a stream, it's set to wide orientation. If you later want to use standard byte i/o functions, you'll need to re-open the stream first. This is probably why the recommendation is not to use it on stdout. However, if you only use wide character functions on stdin and stdout (including any code that you link to), you will not have any problems.
Don't use fputs with anything else than ASCII.
If you want to write down lets say UTF8, then use a function who return the real size used by the utf8 string and use fwrite to write the good number of bytes, without worrying of vicious '\0' inside the string.
I'm reading out text from a website with Curl. All the rawdata is being returned character by character with
return memEof(mp) ? EOF : (int)(*(unsigned char *)(mp->readptr++));
My problem is, that all the special characters such as ÄÖÜäöüß etc are all wrong and look very cryptic. I'm currently correcting them manually by adjusting their values using this table:
http://www.barcoderesource.com/barcodeasciicharacters.shtml
I was wondering now, if there is a more elegant way to do this and how others approach these kinds of issues.
This is an encoding issue. If you read data byte by byte, you can handle correctly and easily just single byte encodings (like ISO-8859 "family" and many more), provided you have a way to convert them correctly in a target encoding, if you need. But with encodings like UTF-8 you are less lucky, since to get the right code you need to read 1 byte, or maybe 2, or maybe three... If you stream them into a string, and print the string altogether, and the output device intended encoding is the same of the input encoding, you get the right char shown anyway.
If it does not happen, and you are not printing each byte as if it were a symbol for sure, then the output device intended encoding does not match the one the string is written with.
If the output, once you print the string "altogether" looks ok, then the problem is that you are interpreting each byte as a single character, while it is not (you have a multibyte encoding for char like the special one you mentioned; likely it is UTF-8 but it could be not too).
If you get equal results in both cases (when you print each byte one by one and when you output a string that keeps the whole byte sequence), then the output device intended encoding is a single byte encoding like the input encoding, but they do not match.
Further details would need to know how you collect the bytes you read in order to print them and say that they looks cryptic.
An example.
const char *string = "\xc3\xa8\xc3\xb2\xc3\xa0";
int i;
for(i = 0; string[i] != 0; i++)
{
printf("%c\n", string[i]);
// using \n is important; if you "sequence" the chars and the output enc is
// utf-8, you obtain the right output
}
printf("%s", string);
You obtain different results if the output device encoding is UTF-8; if it is a single byte encoding, you obtain the same output (newlines apart), but it is "wrong" with respect to what I've written, i.e. èòà.
The "same" text is, in Latin1, "\xe8\xf2\xe0". Latin1 is a single byte encoding, so the above speech applies. If printed on a terminal understanding utf-8, you can obtain something like �� ...
So, encodings matter, device/format output encoding matters too, and you must be aware of both in order to handle and show properly the text. (About format, an example could be html, where you can specify the encoding of the content... you must be coherent, and you'll see everything fine)
I guess you have to use an external library like iconv to create a wchar_t string which contains the data. This depends on the used character encoding.