Cannot correctly print extended ASCII characters in NCURSES - c

I have this program:
#include <ncurses.h>
int main () {
initscr();
mvaddstr(0, 0, " A B C D E ");
mvaddstr(1, 24, "ñandñ");
mvaddstr(1, 34, "esdrñjulñ");
refresh();
getch();
endwin();
return 0;
}
When ncurses prints the first word (ñandñ) something happens that when moving to another position afterwards (in this case to 1, 34), actually it moves to another position, so the second word gets printed in another column.
So what it should look like:
A B C D E
ñandñ esdrñjulñ
looks like this:
A B C D E
ñandñ esdrñjulñ
because of the two 'ñ' extended ascii char in the first word.
Any idea what is wrong?
Thanks!

If you want to use multibyte UTF-8 characters, you must use the version of the ncurses library compiled with multibyte support, and you need to set the locale correctly at the beginning of your program.
The multibyte ncurses library is usually called libncursesw, so to use it is is sufficient to change -lncurses to -lncursesw in your linker options. You do not need a different header file. However, if you actually want to use the wide-character functions, you must #define the symbol _XOPEN_SOURCE_EXTENDED before any #include directive. The easiest way to do that with gcc (or clang) is to add -D_XOPEN_SOURCE_EXTENDED to your compiler options.
If your shell's locale has been set to a UTF-8 locale (which will usually be the case on a modern Linux distribution), it is sufficient to insert
setlocale(LC_ALL, "");
before any call to an ncurses routine. That sets the executable's locale to the locale configured by the environment variables. You'll need to add
#include <locale.h>
to your header includes. No special library needs to be linked for locale support.
The original question indicated that mvaddwstr was being used. That's generally a better solution for multibyte characters, but as indicated above, you can use the narrow string interfaces if you want to. However, you cannot output incomplete UTF-8 sequences, so the single-character narrow interfaces like addch can only be used with character codes less than 128.
This note applied to an attempt to call mvaddwstr with a char* instead of a wchar_t* argument. Don't do that:
Like all of the w ncurses functions which accept strings, mvaddwstr takes a wchar_t* as its string argument. Your compiler should have warned you about that (unless it warned you that there was no prototype for mvaddwstr). So the strings should be prefixed with the L length attribute: L"ñandñ".

Related

Weird space when using wide characters in c

I am trying to draw a square with a given width and height.
I am trying to do so while using the box characters from Unicode.
I am using this code:
#include <stdlib.h>
#include <stdio.h>
#include <wchar.h>
#include <locale.h>
#include "string_prints.h"
#define VERTICAL_PIPE L"║"
#define HORIZONTAL_PIPE L"═"
#define UP_RIGHT_CORNER L"╗"
#define UP_LEFT_CORNER L"╔"
#define DOWN_RIGHT_CORNER L"╝"
#define DOWN_LEFT_CORNER L"╚"
// Function to print the top line
void DrawUpLine(int w){
setlocale(LC_ALL, "");
wprintf(UP_LEFT_CORNER);
for (int i = 0; i < w; i++)
{
wprintf(HORIZONTAL_PIPE);
}
wprintf(UP_RIGHT_CORNER);
}
// Function to print the sides
void DrawSides(int w, int h){
setlocale(LC_ALL, "");
for (int i = 0; i < h; i++)
{
wprintf(VERTICAL_PIPE);
for (int j = 0; j < w; j++)
{
putchar(' ');
}
wprintf(VERTICAL_PIPE);
putchar('\n');
}
}
// Function to print the bottom line
void DrawDownLine(int w){
setlocale(LC_ALL, "");
wprintf(DOWN_LEFT_CORNER);
for (int i = 0; i < w; i++)
{
wprintf(HORIZONTAL_PIPE);
}
wprintf(DOWN_RIGHT_CORNER);
}
void DrawFrame(int w, int h){
DrawUpLine(w);
putchar('\n');
DrawSides(w, h);
putchar('\n');
DrawDownLine(w);
}
But when I am running this code with some int values I get an output with seemingly random spaces and newlines (although the pipes seem at the correct order).
It is being called from main.c from the header like so:
#include <stdlib.h>
#include <stdio.h>
#include <wchar.h>
#include <locale.h>
#include "string_prints.h"
int main(){
DrawFrame(10, 20); // Calling the function
return 0;
}
Also as you can see I don't understand the correct use of setlocale, do you need to do it only once? or more?
Any help appreciated thanks in advance!
Also as you can see I don't understand the correct use of setlocale, do you need to do it only once? or more?
Locale changes applied via setlocale() are persistent within the calling process. You do not need to call that function multiple times unless you want to make multiple changes. But you do need to name a locale to it that serves your intended purpose, or if you call it with an empty string then you or the program user does need to ensure that the environment variables that define the various locale categories are set to values that suit the purpose.
But when I am running this code with some int values I get an output
with seemingly random spaces and newlines.
That sounds like the result of a character-encoding mismatch, or even two (but see also below):
there can be a runtime mismatch because the locale you tell the program to use for output does not match the one expected by the output device (e.g. a terminal) with which the program's output is displayed, and
there can also be a compile time mismatch between the actual character encoding of your source file and the encoding the compiler interprets it as having.
Additionally, use of wide string literal syntax notwithstanding, it is implementation-dependent which characters other than C's basic set may appear in your source code. The wide syntax specifies mostly the form of the storage for the literal (elements of type wchar_t), not so much what character values are valid or how they are interpreted.
Note also that the width of wchar_t is implementation-dependent, and it can be as small as eight bits. It is not necessarily the case that a wchar_t can represent arbitrary Unicode characters -- in fact, it is pretty common for wchar_t to be 16 bits wide, which in fact isn't wide enough for the majority of characters from Unicode's 21-bit code space. You might get an internal representation of wider characters in a two-unit form, such as a UTF-16 surrogate pair, but you also might not -- a great deal of this is left to individual implementations.
Among those things, what encoding the compiler expects, under what circumstances, and how you can influence that are all implementation-dependent. For GCC, for instance, the default source ("input") character set is UTF-8, and you can define a different one via its -finput-charset option. You can also specify both a standard and a wide execution character set via the -fexec-charset and -fwide-exec-charset options, if you wish to do so. GCC relies on iconv for conversions, both at compile time (source charset to execution charset) and at runtime (from execution charset to locale charset). Other implementations have other options (or none), with their own semantics.
So what should you do? In the first place, I suggest taking the source character set out of the equation by using UTF-8 string literals expressed using only basic character set (requires C2011):
#define VERTICAL_PIPE u8"\xe2\x95\x91"
#define HORIZONTAL_PIPE u8"\xe2\x95\x90"
#define UP_RIGHT_CORNER u8"\xe2\x95\x97"
#define UP_LEFT_CORNER u8"\xe2\x95\x94"
#define DOWN_RIGHT_CORNER u8"\xe2\x95\x9d"
#define DOWN_LEFT_CORNER u8"\xe2\x95\x9a"
Note well that the resulting strings are normal, not wide, so you should not use the wide-oriented output functions with them. Instead, use the normal printf, putchar, etc..
And that brings us to another issue with your code: you must not mix wide-oriented and byte-oriented functions writing to the same stream without taking explicit measures to switch (freopen or fwide; see paragraph 7.21.2/4 of the standard). In practice, mixing the two can quite plausibly produce mangled results.
Then also ensure that your local environment variables are set correctly for your actual environment. Chances are good that they already are, but it's worth a check.

What encoding are character / string literals stored in? (Or how to find a literal character in a string from input?)

As we know, different encodings map different representations to same characters. Using setlocale we can specify the encoding of strings that are read from input, but does this apply to string literals as well? I'd find this surprising since these are compile-time!
This matters for tasks as simple as, for example, determining whether a string read from input contains a specific character. When reading strings from input it seems sensible to set the locale to to the user's locale (setlocale("LC_ALL", "");) so that the string is read and processed correctly. But when we're comparing this string with a character literal, won't problems arise due to mismatched encoding?
In other words: The following snippet seems to work for me. But doesn't it work only because of coincidence? Because - for example? - the source code happened to be saved in the same encoding that is used on the machine during runtime?
#include <stdio.h>
#include <wchar.h>
#include <stdlib.h>
#include <locale.h>
int main()
{
setlocale(LC_ALL, "");
// Read line and convert it to wide string so that wcschr can be used
// So many lines! And that's even though I'm omitting the necessary
// error checking for brevity. Ah I'm also omitting free's
char *s = NULL; size_t n = 0;
getline(&s, &n, stdin);
mbstate_t st = {0}; const char* cs = s;
size_t wn = mbsrtowcs(NULL, &cs, 0, &st);
wchar_t *ws = malloc((wn+1) * sizeof(wchar_t));
st = (mbstate_t){0};
mbsrtowcs(ws, &cs, (wn+1), &st);
int contains_guitar = (wcschr(ws, L'🎸') != NULL);
if(contains_guitar)
printf("Let's rock!\n");
else
printf("Let's not.\n");
return 0;
}
How to do this correctly?
Using setlocale we can specify the encoding of strings that are read from input, but does this apply to string literals as well?
No. String literals use the execution character set, which is defined by your compiler at compile time.
Execution character set does not have to be the same as the source character set, the character set used in the source code. The C compiler is responsible for the translation, and should have options for choosing/defining them. The default depends on the compiler, but on Linux and most current POSIXy systems, is usually UTF-8.
The following snippet seems to work for me. But doesn't it work only because of coincidence?
The example works because the character set of your locale, the source character set, and the execution character set used when the binary was constructed, all happen to be UTF-8.
How to do this correctly?
Two options. One is to use wide characters and string literals. The other is to use UTF-8 everywhere.
For wide input and output, see e.g. this example in another answer here.
Do note that getwline() and getwdelim() are not in POSIX.1, but in C11 Annex K. This means they are optional, and as of this writing, not widely available at all. Thus, a custom implementation around fgetwc() is recommended instead. (One based on fgetws(), wcslen(), and/or wcscspn() will not be able to handle embedded nuls, L'\0', correctly.)
In a typical wide I/O program, you only need mbstowcs() to convert command-line arguments and environment variables to wide strings.
Using UTF-8 everywhere is also a perfectly valid practical approach, at least if it is well documented, so that users know the program inputs and outputs UTF-8 strings, and developers know to ensure their C compiler uses UTF-8 as the execution character set when compiling those binaries.
Your program can even use e.g.
if (!setlocale(LC_ALL, ""))
fprintf(stderr, "Warning: Your C library does not support your current locale.\n");
if (strcmp("UTF-8", nl_langinfo(CODESET)))
fprintf(stderr, "Warning: Your locale does not use the UTF-8 character set.\n");
to verify the current locale uses UTF-8.
I have used both approaches, depending on the circumstances. It is difficult to say which one is more portable in practice, because as usual, both work just fine on non-Windows OSes without issues.
If you're willing to assume UTF-8,
strstr(s,"🎸")
Or:
strstr(s,u8"🎸")
The latter avoids some assumptions but requires a C11 compiler. If you want the best of both and can sacrifice readability:
strstr(s,"\360\237\216\270")

Use the letter ñ in C

I have to save in a char[] the letter ñ and I'm not being able to do it. I tried doing this:
char example[1];
example[0] = 'ñ';
When compiling I get this:
$ gcc example.c
error: character too large for enclosing
character literal type
example[0] = 'ñ';
Does anyone know how to do this?
If you're using High Sierra, you are presumably using a Mac running macOS 10.13.3 (High Sierra), the same as me.
This comes down to code sets and locales — and can get tricky. Mac terminals use UTF-8 by default and ñ is Unicode character U+00F1, which requires two bytes, 0xC3 and 0xB1, to represent it in UTF-8. And the compiler is letting you know that one byte isn't big enough to hold two bytes of data. (In the single-byte code sets such as ISO 8859-1 or 8859-15, ñ has character code 0xF1 — 0xF1 and U+00F1 are similar, and this is not a coincidence; Unicode code points U+0000 to U+00FF are the same as in ISO 8859-1. ISO 8859-15 is a more modern variant of 8859-1, with the Euro symbol € and 7 other variations from 8859-1.)
Another option is to change the character set that your terminal works with; you need to adapt your code to suit the code set that the terminal uses.
You can work around this by using wchar_t:
#include <wchar.h>
void function(void);
void function(void)
{
wchar_t example[1];
example[0] = L'ñ';
putwchar(example[0]);
putwchar(L'\n');
}
#include <locale.h>
int main(void)
{
setlocale(LC_ALL, "");
function();
return 0;
}
This compiles; if you omit the call to setlocale(LC_ALL, "");, it doesn't work as I want (it generates just octal byte \361 (aka 0xF1) and a newline, which generates a ? on the terminal), whereas with setlocale(), it generates two bytes (\303\261 in octal, aka 0xC3 and 0xB1) and you see ñ on the console output.
You can use "extended ascii". This chart shows that 'ñ' can be represented in extended ascii as 164.
example[0] = (char)164;
You can print this character just like any other character
putchar(example[0]);
As noted in the comments above, this will depend on your environment. It might work on your machine but not another one.
The better answer is to use unicode, for example:
wchar_t example = '\u00F1';
This really depends on which character set / locale you will be using. If you want to hardcode this as a latin1 character, this example program does that:
#include <cstdio>
int main() {
char example[2] = {'\xF1'};
printf("%s", example);
return 0;
}
This, however, results in this output on my system that uses UTF-8:
$ ./a.out
�
So if you want to use non-ascii strings, I'd recommend not representing them as char arrays directly. If you really need to use char directly, the UTF-8 sequence for ñ is two chars wide, and can be written as such (again with a terminating '\0' for good measure):
char s[3] = {"\xC3\xB1"};

Greek Character Conversion

I am trying to make a simple -ancient greek to modern greek- converter, in c, by changing the tones of the vowels. For example, the user types a text in greek which conains the character: ῶ (unicode: U+1FF6), so the program converts it into: ώ (unicode:U+1F7D). Greek are not sopported by c, so I don't know how to make it work. Any ideas?
Assuming you use a sane operating system (meaning, not Windows), this is very easy to achieve using C99/C11 locale and wide character support. Consider filter.c:
#include <stdlib.h>
#include <locale.h>
#include <wchar.h>
#include <stdio.h>
wint_t convert(const wint_t wc)
{
switch (wc) {
case L'ῶ': return L'ώ';
default: return wc;
}
}
int main(void)
{
wint_t wc;
if (!setlocale(LC_ALL, "")) {
fprintf(stderr, "Current locale is unsupported.\n");
return EXIT_FAILURE;
}
if (fwide(stdin, 1) <= 0) {
fprintf(stderr, "Standard input does not support wide characters.\n");
return EXIT_FAILURE;
}
if (fwide(stdout, 1) <= 0) {
fprintf(stderr, "Standard output does not support wide characters.\n");
return EXIT_FAILURE;
}
while ((wc = fgetwc(stdin)) != WEOF)
fputwc(convert(wc), stdout);
return EXIT_SUCCESS;
}
The above program reads standard input, converts each ῶ into a ώ, and outputs the result.
Note that wide character strings and characters have an L prefix; L'ῶ' is a wide character constant. These are only in Unicode if the execution character set (the character set the code is compiled for) is Unicode, and that depends on your development environment. (Fortunately, outside of Windows, UTF-8 is pretty much a standard nowadays -- and that is a good thing -- so code like the above Just Works.)
On POSIXy systems (like Linux, Android, Mac OS, BSDs), you can use the iconv() facilities to convert from any input character set to Unicode, do the conversion there, and finally convert back to any output character set. Unfortunately, the question is not tagged posix, so that is outside this particular question.
The above example uses a simple switch/case statement. If there are many replacement pairs, one could use e.g.
typedef struct {
wint_t from;
wint_t to;
} widepair;
static widepair replace[] = {
{ L'ῶ', L'ώ' },
/* Others? */
};
#define NUM_REPLACE (sizeof replace / sizeof replace[0])
and at runtime, sort replace[] (using qsort() and a function that compares the from elements), and use binary search to quickly determine if a wide character is to be replaced (and if so, to which wide character). Because this is a O(log2N) operation with N being the number of pairs, and it utilizes cache okay, even thousands of replacement pairs is not a problem this way. (And of course, you can build the replacement array at runtime just as well, even from user input or command-line options.)
For Unicode characters, we could use a uint32_t map_to[0x110000]; to directly map each code point to another Unicode code point, but because we do not know whether wide characters are Unicode or not, we cannot do that; we do not know the code range of the wide characters until after compile time. Of course, we can do a multi-stage compilation, where a test program generates the replace[] array shown above, and outputs their codes in decimal; then do some kind of auto-grouping or clustering, for example bit maps or hash tables, to do it "even faster".
However, in practice it usually turns out that the I/O (reading and writing the data) takes more real-world time than the conversion itself. Even when the conversion is the bottleneck, the conversion rate is sufficient for most humans. (As an example, when compiling C or C++ code with the GNU utilities, the preprocessor first converts the source code to UTF-8 internally.)
Okay, here's some quick advice. I wouldn't use C because Unicode is not wel supported (yet).
A better language choice would be Python, Java, ..., anything with good Unicode support.
I'd write a utility that reads from standard input and writes to standard output. This makes it easy to use from the command line and in scripts.
I might be missing something but it's going to be something like this (in pseudo code):
while ((inCharacter = getCharacterFromStandardInput) != EOF
{
switch (inCharacter)
{
case 'ῶ': outCharacter = ώ; break
...
}
writeCharacterToStandardOutput(outCharacter)
}
You'll also need to select & handle the format: UTF-8/16/32.
That's it. Good luck!

wide characters transliteration C with towctrans

I would like to convert (transliterate) UTF-8 characters to be closest match in ASCII in C. Characters like ú is transliterated to u. I can do that with iconv, with iconv -f utf-8 -t ascii//TRANSLIT, on the command line.
In C, there is a function towctrans to do that, but I only found documentation about two possible transliterations: to lower case and to upper case (see man wctrans). On the documentation, wctrans depends on LC_CTYPE. But what other function (other than "tolower" and "toupper") are available for a specific LC_CTYPE value?
A simple example with towctrans and the basic toupper transliteration:
#include <stdio.h>
#include <wchar.h>
#include <locale.h>
int main() {
wchar_t frase[] = L"Amélia"; int i;
setlocale(LC_ALL, "");
for (i=0; i < wcslen(frase); i++) {
printf("%lc --> %lc\n", frase[i], towctrans(frase[i], wctrans("toupper")));
}
}
I know I can do this conversion with libiconv, but I was trying to find out possible already defined wctrans functions.
While the standard allows for implementation-defined or locale-defined transformations via wctrans, I'm not aware of any existing implementations that offer such a feature, and it's certainly not widespread. The iconv approach of //TRANSLIT is also non-standard and in fact conflicts with the standard: POSIX requires a charset name containing a slash character to be interpreted as a pathname to a charmap file, so use of slash for specifying translit-mode is non-conforming.

Resources