Reading and printing out Unicode and UTF-8 encodings in C - c

Question

I'm using Windows XP.
I want to read from files in ASCII, UTF-8 and Unicode encodings and print out strings on stdout.
I was trying to use functions from wchar.h like fgetwc()/fputwc() and fgetws()/fputws(), they work on ASCII but not when a file is in UTF-8 or Unicode. Doesn't print out language specific characters and when a file is in Unicode it doesn't print out anything but the box and first letter.
Is there any way of making a program in pure C that will read files, compare strings and print them out correctly on stdout regardless of the encoding of the files fed to the program?

Answer 1

Since you're on Windows, the key is that you want to write your strings out using the WriteConsoleW function, having first assembled the sequence of UTF-16 characters that you want to write out. (You probably should only write a few kilobytes of characters at a time.) Use GetStdHandle to obtain the console handle, of course.
Harder is determining the encoding of a file. Luckily, you don't need to distinguish between ASCII and UTF-8 as the latter is a strict superset of the former. But for any other single-byte encoding, you need to guess. Some UTF-8 files, more likely so on Windows than elsewhere, have a UTF-8 encoded byte-order mark at the beginning of the file; that's nasty as BOMs are not really supposed to be used with UTF-8, but a strong indicator if present. (Spotting UTF-16 is easier, as it should either have a byte-order mark, or you can guess it from the presence of NUL (0) bytes.)

Answer 2

Here's a little piece of code I used to print various characters outside of the ASCII subset of Unicode (contains workarounds for what seems to be a bug in the Open Watcom compiler's implementation of printf()):
// Compile with Open Watcom C/C++ 1.9: wcl386 cons-utf8.c
#include <windows.h>
#include <stdio.h>
#include <stddef.h>
// Workarounds for printf() not printing multi-byte (UTF-8) strings
// with Open Watcom C/C++ 1.7-1.9.
// 0 - no workaround for printf()
// 1 - setbuf(stdout, NULL) before printf()
// 2 - fflush(stdout) after printf()
// 3 - WriteConsole() instead of printf()
#define PRINT_WORKAROUND 03
int main(void)
{
DWORD err, i, j;
// Code point ranges of characters to print
static const DWORD ranges[][2] =
{
{ 0x0A0, 0x0FF }, // Latin chars with diacritic marks + some others
{ 0x391, 0x3CE }, // Greek chars
{ 0x410, 0x44F } // Cyrillic chars
};
#if PRINT_WORKAROUND == 1
setbuf(stdout, NULL);
#endif
if (!SetConsoleOutputCP(CP_UTF8))
{
err = GetLastError();
printf("SetConsoleOutputCP(CP_UTF8) failed with error 0x%X\n", err);
goto Exit;
}
printf("Workaround: %d\n", PRINT_WORKAROUND);
for (j = 0; j < sizeof(ranges) / sizeof(ranges[0]); j++)
{
for (i = ranges[j][0]; i <= ranges[j][1]; i++)
{
char str[8];
int sz;
wchar_t wstr[2];
wstr[0] = i;
wstr[1] = 0;
sz = WideCharToMultiByte(CP_UTF8,
0,
wstr,
-1,
str,
sizeof(str),
NULL,
NULL);
if (sz <= 0)
{
err = GetLastError();
printf("WideCharToMultiByte() failed with error 0x%X\n", err);
goto Exit;
}
#if PRINT_WORKAROUND < 3
printf("%s", str);
#if PRINT_WORKAROUND == 2
fflush(stdout);
#endif
#else
WriteConsole(GetStdHandle(STD_OUTPUT_HANDLE),
str,
sz - 1,
&err,
NULL);
#endif
}
printf("\n");
}
printf("\n");
Exit:
return 0;
}
Output:
C:\>cons-utf8.exe
Workaround: 3
 ¡¢£¤¥¦§¨©ª«¬­®¯°±²³´µ¶·¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿ
ΑΒΓΔΕΖΗΘΙΚΛΜΝΞΟΠΡ΢ΣΤΥΦΧΨΩΪΫάέήίΰαβγδεζηθικλμνξοπρςστυφχψωϊϋόύώ
АБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯабвгдежзийклмнопрстуфхцчшщъыьэюя
I didn't find a way to print UTF-16 code points directly to the console in Windows XP that would work the same as above.

Answer 3

Is there any way of making a program in pure C that will read files, compare strings and print them out correctly on stdout regardless of the encoding of the files fed to the program?
No, the program has obviously to told the encoding of the files too. Internally, you can choose to represent the data of the files with multibyte strings in UTF-8 or with wide strings.