I need to process some Win-1251-encoded text (8-bit encoding, uses some of 128..255 for Cyrillic). As far as I can tell, C was created with 7-bit ASCII in mind, no explicit support for single-byte chars above 127. So I have several questions:
Which is the more proper type for this text: char[] or unsigned char[]?
If I use unsigned char[] with built-in functions (strlen, strcmp), the compiler warns about implicit casts to char*. Can such a cast break something? Should I re-implement some of the functions to support unsigned char strings explicitly?
C has three distinct character types, signed char, unsigned char, and char, which may be either signed or unsigned. For strings, you should just use char, since that will play nice with all the built-in functions. They'll all also work fine on characters with numeric values greater than 127. You should have no problems with your case using char.
Related
I'm using mplabX 4.20, and xc8 compiler. I'm trying to understand which is the difference between uint8_t and unsigned char. Both of them have size from 0 till 255.
Both of can hold characters and numbers. But which is better to use, and for which case?
Example if i want to create a buffer for holding a string.
uint8_t buffer[20]="Hello World";
unsigned char buffer[20]="Hello World";
In most cases i need to hold characters. Which is the best practise for this action?
I'm using mplabX 4.20, and xc8 compiler. I'm trying to understand
which is the difference between uint8_t and unsigned char. Both of
them have size from 0 till 255. Both of can hold characters and
numbers. But which is better to use, and for which case?
unsigned char is the unsigned integer type corresponding to signed char. Its representation does not use any padding bits. Both of these occupy the same amount of storage as type char, which is at least 8 bits, but may be more. The macro CHAR_BIT tells you how many it comprises in your implementation. Every conforming C implementation provides all of these types.
uint8_t, if available, is an unsigned integer data type exactly 8 bits wide and with no padding bits. On an implementation having CHAR_BIT defined as 8, this is the same type as unsigned char. On such systems you may use the two types interchangeably wherever the declarations provided by stdint.h are in scope. On other systems, uint8_t will not be declared at all.
Example if i want to create a buffer for holding a string.
If you want to declare a buffer for holding a string then as a matter of style, you should use type char, not either of the other two:
char buffer[20] = "Hello World";
Although either of the other two, or signed char, can also be used for string data (provided in the case of uint8_t that the type is defined at all), type char is the conventional one to use for character data. Witness, for example, that that's the type in terms of which all the string.h functions are declared.
You should use uint8_t where and only where you need an integer type with exactly its properties: unsigned, 8 value bits, no padding bits.
You should use unsigned char where you want the smallest unsigned integer type available, but you don't care whether it is exactly 8 bits wide, or where you want to emphasize that it is the same size as a char -- the smallest discrete unit of storage available.
You should use signed char where you want the smallest signed integer type available but don't care about the exact size or representation.
You should use int8_t where you want a signed integer type with exactly 7 value bits, one sign bit, and no padding bits, expressed in two's complement representation.
You should remain mindful that uint8_t and int8_t are not guaranteed to be available from every C implementation, and that where they are available, their use requires inclusion of stdint.h. Furthermore, this header and these types were not part of C90 at all, so you should not use them if compatibility with legacy C implementations is important to you.
difference between uint8_t and unsigned char
If you're on a some exotic system where CHAR_BIT > 8, then uint8_t isn't going to be defined at all.
Otherwise (if CHAR_BIT == 8) there is no difference between unsigned char and uint8_t.
i need to hold characters
Then use plain char.
Functions that operate in strings usually have [const]char * parameters, and you won't be able to pass your unsigned char arrays to them.
This question:
What is an unsigned char?
does a great job of discussing char vs. unsigned char vs. signed char in C.
However, it doesn't directly address what should be used for non-ASCII text. Thus if I have an array of bytes that represents text in some arbitrary character set like UTF-8 or Big5 (or sometimes ASCII), should I use an array of char or unsigned char?
I'm leaning towards using char because otherwise gcc gives me warnings about signedness of pointers when the array is ASCII and I use strlen. But I would like to know what is correct.
Use normal char to represent characters. Use signed char when you want a signed integer type that covers values from -127 to +127 . Use unsigned char for having an unsigned integer type that has range of values from 0 to 255 .
The question you are asking is probably much broader that you expect.
To answer it directly, most implementations use "byte" as underlying buffer. In that terms standard uint8_t typedef is your best bet. That is primarily because most character sets use variable number of bytes to store characters, so separate byte processing is essential in encoding and decoding process. It also simplifies conversion between different "endianess".
In general it's incorrect to use strlen on anything other than ASCII encoding or other single-byte code pages (0-255 range). It's certainly incorrect on any multi-byte encoding like Big5, UTF-8/16 or Shift-JIS.
As far as UTF8 or any encoding where ASCII characters have the same codepoints, char is the best type for multi-byte characters string:
assume typedef char utf8:
This is the only way to allow char * to be used as utf8 * without an explicit cast. This is extremely common and a good enough reason to be better than unsigned char.
utf8 * could be accidentally passed to function expecting a pointer to a sequence of ASCII characters, but this could also be needed if you need to printf your utf8 string (which is a valid thing to do)
The main drawback is that as char sign is unknown, usage of arithmetic operators like > is unsafe, and the only safe way to check if a character is in the ASCII range is by checking the bit directly with ISASCII(c) ((c & (1 << 7) == 0)
What is the need for signed and unsigned characters in C?
Is there some special reason for having a signed and unsigned char in C? Or was it simply added for completeness so that the compiler does not have to check the data type before adding signed/unsigned modifier?
I am not asking about signed and unsigned variables. My doubt is about the special cases where an unsigned character variable will not be sufficient such that you have to depend on a signed character variable.
A char can be either signed or unsigned depending on what is most efficient for the underlying hardware. The keywords signed and unsigned allow you to explicitly specify that you want something else.
A quote from the C99 rationale:
Three types of char are specified: signed, plain, and unsigned. A plain char may be represented as either signed or unsigned depending upon the implementation, as in prior practice. The type signed char was introduced in C89 to make available a one-byte signed integer type on those systems which implemented plain char as unsigned char. For reasons of symmetry, the keyword signed is allowed as part of the type name of other integer types.
Information #1: char in C is just a small int, which uses 8 bits.
Information #2: Difference between signed and unsigned, is that one bit in the representation is used as the sign bit for a signed variable.
Information #3: As a result of (#2), signed variables hold different ranges (-128 to 127, in char case) compared to unsigned (0 to 255 in char case).
Q-A #1: why do we need unsigned?
In most cases (for instance representing a pointer) we do not need signed variables. By convention all locations in the memory are exposed to the program as a contiguous array of unsigned addresses.
Q-A #2: why do we need signed?
Generally, to do signed arithmetic.
I assume you are using a char to hold numbers, not characters.
So:
signed char gives you at least the -128 to 127 range.
unsigned char gives you at least the 0 to 255 range.
A char is required by standard to be AT LEAST 8 bits, so that is the reason for my saying at least. It is possible for these values to be larger.
Anyway, to answer your question, having a char as unsigned frees the requirement for the first bit to be the 'sign' bit, thus allowing you to hold near double that of a signed char.
The thing you have to understand is that datatype "char" is actually just an integer, typically 8-bits wide. You can use it like any other inter datatype, assuming you respect the reduced value limits. There is no reason to limit "char" to characters.
On a 32/64-bit processor, there is typically no need to use such small integer fields, but on an 8-bit processor such as the 8051, 8-bit integers are no only much faster to process and use less (limited) memory.
Given unsigned char *str, a UTF-8 encoded string, is it legal to write the first byte (not character) with fputc((char)(*str), file);
Remove the cast to char. fputc takes the character to write as an int argument whose value is expected to be in the range of unsigned char, not char. Assuming (unsigned char)(char) acts as the identity operator on unsigned char values, there's no error in your code, but it's not guaranteed to especially for oddball systems without twos complement.
It's legal. fputc converts its int input to unsigned char, and that conversion can't do anything too unpleasant. It just takes the value modulo UCHAR_MAX+1.
If char is unsigned on your implementation, then converting from unsigned char to char doesn't affect the value.
If char is signed on your implementation, then converting a value greater than CHAR_MAX to char either has an implementation-defined result, or raises a signal (6.3.1.3/3). So while your code is legal, the possible behavior includes raising a signal that terminates the program, which might not be what you want.
In practice, you expect implementations to use 2's complement, and to convert to signed types in the "obvious" way, preserving the bit pattern.
Even if nothing else goes wrong, your terminal might not do anything sensible, if you write a strange byte to STDOUT.
No, you have to pass a FILE pointer as second parameter. This is the file handle you would like to write the character to, for example stdout.
fputc(*str, stdout);
Yes it is legal. fputc will just write a byte. The cast to signed/unsigned in this case will just stop the compiler moaning at you.
From The C Programming Language (Brian W. Kernighan), 2.7 TYPE CONVERSIONS, pg 43 :
"There is one subtle point about the
conversion of characters to integers.
... On some macines a char whose
leftmost bit is 1 will be converted to
a negative integer. On others, ... is
always positive. For portability,
specify signed or unsigned if
non-character data is to be stored in
char variables."
My questions are:
Why would anyone want to store
non-char data in char? (an example
where this is necessary will be real
nice)
Why does integer value of char
change when it is converted to int?
Can you elaborate more on this
portability issue?
In regards to 1)
People often use char arrays when they really want a byte buffer for a data stream. Its not great practice, but plenty of projects do it, and if you're careful, no real harm is done. There are probably other times as well.
In regards to 2)
Signed integers are often sign extended when they are moved from a smaller data type. Thus
11111111b (-1 in base 10) becomes 11111111 11111111 11111111 11111111 when expanded to 32 bits. However, if the char was intended to be unsigned +255, then the signed integer may end up being -1.
About portability 3)
Some machines regard chars as signed integers, while others interpret them as unsigned. It could also vary based on compiler implementation. Most of the time you don't have to worry about it. Kernighan is just trying to help you understand the details.
Edit
I know this is a dead issue, but you can use the following code to check if char's on your system are signed or unsigned:
#include <limits.h> //Include implementation specific constants (MAX_INT, et c.)
#if CHAR_MAX == SCHAR_MAX
// Plain "char" is signed
#else
// Plain "char" is unsigned
#endif
1) char is the size of a single byte in C, and is therefore used for storing any sort of data. For example, when loading an image into memory, the data is represented as an array of char. In modern code, typedefs such as uint8_t are used to indicate the purpose of a buffer more usefully than just char.
2 & 3) Whether or not char is signed or unsigned is platform dependent, so if a program depends on this behavior then it's best to specify one or the other explicitly.
The char type is defined to hold one byte, i.e. sizeof(char) is defined to be 1. This is useful for serializing data, for instance.
char is implementation-defined as either unsigned char or signed char. Now imagine that char means smallint. You are simply converting a small integer to a larger integer when you go from smallint to int. The problem is, you don't know whether that smallint is signed or unsigned.
I would say it's not really a portability issue as long as you follow The Bible (K&R).
unsigned char is often used to process binary data one byte at a time. A common example is UTF-8 strings, which are not strictly made up of "chars."
If a signed char is 8 bits and the top bit is set, that indicates that it's negative. When this is converted to a larger type, the sign is kept by extending the high bit to the high bit of the new type. This is called a "sign-extended" assignment.
1) Char is implemented as one byte across all systems so it is consistent.
2) The bit mentioned in you question is the one that is used in single byte integers for their singed-ness. When a int on a system is larger than one byte the signed flat is not affected when you convert char to int, other wise it is. ( there are also singed and unsigned chars)
3) Because of the consistence of the char implementation lots of libs use them like the Intel IPP (Intel Performance Primitives) libs and their cousins OpenCV.
Usually, in C, char to int conversion and vice versa is an issue because the stanard APIs for reading character input/writing character output use int's for the character arguments and return values. See getchar(), getc() and putchar() for example.
Also, since the size of a char is 1 byte, it is a convenient way to deal with arbitrary data as a byte stream.