I am confused as to how C pointers actually reference the memory address of a variable. I am probably missing something here, but if, for example an int is 32 bits (like in C), then this would be stored in 4 bytes.
If I am not mistaken then each memory address tends to be a byte in size, as these are generally the smallest units of addressable memory. So if an int takes up 4 bytes, then wouldn't it have 4 memory addresses? (as it is stored over 4 8-bit memory addresses).
If this is the case, then how come a pointer only holds one memory address? (or rather only displays one when printed, if it holds more?). Is this simply the first address that stores the int? (assuming they are stored contiguously).
I have tried to find answers online but this has only led to further confusion.
Yes, technically, there would be four addressable bytes for the int you describe. But the pointer points to the first byte, and reading an int from it reads that byte and the subsequent three bytes to construct the int value.
If you tried to read from a pointer referring to one of the other three bytes, at the very least you'd get a different value (because it would read the remains of the one int, and additional bytes next to it), and on some architectures which require aligned reads (so four byte values must begin at an address divisible by four), your program could crash.
The language tries to protect you from reading a misaligned pointer like that; if you have an int*, and add 1 to it, it doesn't increment the raw address by one, it increments it by sizeof(int) (for your case, 4), so that pointers to arrays of int can traverse the array value by value without accidentally reading a value that's logically partially from one int, and partially from its neighbor.
Pointer points to the starting address of your type, if you google "pointer size" it will show you it is generally dependent to your cpu architecture, not to your primitive type or object.
What is the size of a pointer?
which will hopefully support your thoughts although the question is about c++
One byte is the smallest addressable unit, but that doesn't mean an address is only one byte. Otherwise you'd only have 256 bytes you could address! Pointers are typically either 4 or 8 bytes on size
The address of a variable refers to the address of it's first byte. The remaining bytes are understood to immediately follow those, and the number of bytes are part of the datatype.
The specifics depend on the actual architecture of the machine (what kind a CPU, what kind of memory, etc) so I am assuming you care about a modern 32 bit processor where both an int and a pointer take four bytes of memory. Keep in mind that the same ideas apply when integers are two bytes and when pointers are 8 bytes but we have to focus on just one set of examples.
Having said all that, you are completely correct that an int uses four contiguous bytes of memory which means it has four separate memory addresses and that a pointer holds only one address - it is the address of the first byte of the int.
So the CPU has an instruction for reading an int. The instruction takes the address of the first byte of the int and reads an entire int - all four of them. And that's why you only need one address to read an entire int. So int i = 42 reads a four byte integer into i and the value is interpreted to mean the number 42.
But a pointer is also an integer where the value is a memory address, so it can be read exactly the same way. So int *p = 42 reads a four byte integer into p and the value is interpreted to mean memory address 42.
All this gets complicated when you start taking about the order the bytes are stored in so we won't talk about that (however, if you want to find out the term is endianness - see https://en.wikipedia.org/wiki/Endianness)
Related
I'm trying to get a better understanding of the C standard. In particular I am interested in how pointer arithmetic might work in an implementation for an unusual machine architecture.
Suppose I have a processor with 64 bit wide registers that is connected to RAM where each address corresponds to a cell 8 bits wide. An implementation for C for this machine defines CHAR_BIT to be equal to 8. Suppose I compile and execute the following lines of code:
char *pointer = 0;
pointer = pointer + 1;
After execution, pointer is equal to 1. This gives one the impression that in general data of type char corresponds to the smallest addressable unit of memory on the machine.
Now suppose I have a processor with 12 bit wide registers that is connected to RAM where each address corresponds to a cell 4 bits wide. An implementation of C for this machine defines CHAR_BIT to be equal to 12. Suppose the same lines of code are compiled and executed for this machine. Would pointer be equal to 3?
More generally, when you increment a pointer to a char, is the address equal to CHAR_BIT divided by the width of a memory cell on the machine?
Would pointer be equal to 3?
Well, the standard doesn't say how pointers are implemented. The standard tells what is to happen when you use a pointer in a specific way but not what the value of a pointer shall be.
All we know is that adding 1 to a char pointer, will make the pointer point at the next char object - where ever that is. But nothing about pointers value.
So when you say that
pointer = pointer + 1;
will make the pointer equal 1, it's wrong. The standard doesn't say anything about that.
On most systems a char is 8 bit and pointers are (virtual) memory addresses referencing a 8 bit addressable memory loacation. On such systems incrementing a char pointer will increase the pointer value (aka memory address) by 1. However, on - unusual architectures - there is no way to tell.
But if you have a system where each memory address references 4 bits and a char is 12 bits, it seems a good guess that ++pointer will increase the pointer by three.
Pointers are incremented by the minimum of they width of the datatype they "point to", but are not guaranteed to increment to that size exactly.
For memory alignment purposes, there are many times where a pointer might increment to the next memory word alignment past the minimum width.
So, in general, you cannot assume this pointer to be equal to 3. It very well may be 3, 4, or some larger number.
Here is an example.
struct char_three {
char a;
char b;
char c;
};
struct char_three* my_pointer = 0;
my_pointer++;
/* I'd be shocked if my_pointer was now 3 */
Memory alignment is machine specific. One cannot generalize about it, except that most machines define a WORD as the first address that can be aligned to a memory fetch on the bus. Some machines can specify addresses that don't align with the bus fetches. In such a case, selecting two bytes that span the alignment may result in loading two WORDS.
Most systems don't accept WORD loads on non-aligned boundaries without complaining. This means that a bit of boiler plate assembly is applied to translate the fetch to the proceeding WORD boundary, if maximum density is desired.
Most compilers prefer speed to maximum density of data, so they align their structured data to take advantage of WORD boundaries, avoiding the extra calculations. This means that in many cases, data that is not carefully aligned might contain "holes" of bytes that are not used.
If you are interested in details of the above summary, you can read up on Data Structure Alignment which will discuss alignment (and as a consequence) padding.
char *pointer = 0;
After execution, pointer is equal to 1
Not necessarily.
This special case gives you a null pointer, since 0 is a null pointer constant. Strictly speaking, such a pointer is not supposed to point at a valid object. If you look at the actual address stored in the pointer, it could be anything.
Null pointers aside, the C language expects you to do pointer arithmetic by first pointing at an array. Or in case of char, you can also point at a chunk of generic data such as a struct. Everything else, like your example, is undefined behavior.
An implementation of C for this machine defines CHAR_BIT to be equal to 12
The C standard defines char to be equal to a byte, so your example is a bit weird and contradicting. Pointer arithmetic will always increase the pointer to point at the next object in the array. The standard doesn't really speak of representation of addresses at all, but your fictional example that would sensibly increase the address by 12 bits, because that's the size of a char.
Fictional computers are quite meaningless to discuss even from a learning point-of-view. I'd advise to focus on real-world computers instead.
When you increment a pointer to a char, is the address equal to CHAR_BIT divided by the width of a memory cell on the machine?
On a "conventional" machine -- indeed on the vast majority of machines where C runs -- CHAR_BIT simply is the width of a memory cell on the machine, so the answer to the question is vacuously "yes" (since CHAR_BIT / CHAR_BIT is 1.).
A machine with memory cells smaller than CHAR_BIT would be very, very strange -- arguably incompatible with C's definition.
C's definition says that:
sizeof(char) is exactly 1.
CHAR_BIT, the number of bits in a char, is at least 8. That is, as far as C is concerned, a byte may not be smaller than 8 bits. (It may be larger, and this is a surprise to many people, but it does not concern us here.)
There is a strong suggestion (if not an explicit requirement) that char (or "byte") is the machine's "minimum addressable unit" or some such.
So for a machine that can address 4 bits at a time, we would have to pick unnatural values for sizeof(char) and CHAR_BIT (which would otherwise probably want to be 2 and 4, respectively), and we would have to ignore the suggestion that type char is the machine's minimum addressable unit.
C does not mandate the internal representation (the bit pattern) of a pointer. The closest a portable C program can get to doing anything with the internal representation of a pointer value is to print it out using %p -- and that's explicitly defined to be implementation-defined.
So I think the only way to implement C on a "4 bit" machine would involve having the code
char a[10];
char *p = a;
p++;
generate instructions which actually incremented the address behind p by 2.
It would then be an interesting question whether %p should print the actual, raw pointer value, or the value divided by 2.
It would also be lots of fun to watch the ensuing fireworks as too-clever programmers on such a machine used type punning techniques to get their hands on the internal value of pointers so that they could increment them by actually 1 -- not the 2 that "proper" additions of 1 would always generate -- such that they could amaze their friends by accessing the odd nybble of a byte, or confound the regulars on SO by asking questions about it. "I just incremented a char pointer by 1. Why is %p showing a value that's 2 greater?"
Seems like the confusion in this question comes from the fact that the word "byte" in the C standard doesn't have the typical definition (which is 8 bits). Specifically, the word "byte" in the C standard means a collection of bits, where the number of bits is specified by the implementation-defined constant CHAR_BITS. Furthermore, a "byte" as defined by the C standard is the smallest addressable object that a C program can access.
This leaves open the question as to whether there is a one-to-one correspondence between the C definition of "addressable", and the hardware's definition of "addressable". In other words, is it possible that the hardware can address objects that are smaller than a "byte"? If (as in the OP) a "byte" occupies 3 addresses, then that implies that "byte" accesses have an alignment restriction. Which is to say that 3 and 6 are valid "byte" addresses, but 4 and 5 are not. This is prohibited by section 6.2.8 which discusses the alignment of objects.
Which means that the architecture proposed by the OP is not supported by the C specification. In particular, an implementation may not have pointers that point to 4-bit objects when CHAR_BIT is equal to 12.
Here are the relevant sections from the C standard:
§3.6 The definition of "byte" as used in the standard
[A byte is an] addressable unit of data storage large enough to hold
any member of the basic character set of the execution environment.
NOTE 1 It is possible to express the address of each individual byte
of an object uniquely.
NOTE 2 A byte is composed of a contiguous sequence of bits, the number
of which is implementation-defined. The least significant bit is
called the low-order bit; the most significant bit is called the
high-order bit.
§5.2.4.2.1 describes CHAR_BIT as the
number of bits for smallest object that is not a bit-field (byte)
§6.2.6.1 restricts all objects that are larger than a char to be a multiple of CHAR_BIT bits:
[...]
Except for bit-fields, objects are composed of contiguous sequences of
one or more bytes, the number, order, and encoding of which are either
explicitly specified or implementation-defined.
[...] Values stored in non-bit-field objects of any other object type
consist of n × CHAR_BIT bits, where n is the size of an object of that
type, in bytes.
§6.2.8 restricts the alignment of objects
Complete object types have alignment requirements which place
restrictions on the addresses at which objects of that type may be
allocated. An alignment is an implementation-defined integer value
representing the number of bytes between successive addresses at which
a given object can be allocated.
Valid alignments include only those values returned by an _Alignof
expression for fundamental types, plus an additional
implementation-defined set of values, which may be empty. Every
valid alignment value shall be a nonnegative integral power of two.
§6.5.3.2 specifies the sizeof a char, and hence a "byte"
When sizeof is applied to an operand that has type char, unsigned
char, or signed char, (or a qualified version thereof) the result is
1.
The following code fragment demonstrates an invariant of C pointer arithmetic -- no matter what CHAR_BIT is, no matter what the hardware least addressable unit is, and no matter what the actual bit representation of pointers is,
#include <assert.h>
int main(void)
{
T x[2]; // for any object type T whatsoever
assert(&x[1] - &x[0] == 1); // must be true
}
And since sizeof(char) == 1 by definition, this also means that
#include <assert.h>
int main(void)
{
T x[2]; // again for any object type T whatsoever
char *p = (char *)&x[0];
char *q = (char *)&x[1];
assert(q - p == sizeof(T)); // must be true
}
However, if you convert to integers before performing the subtraction, the invariant evaporates:
#include <assert.h>
#include <inttypes.h>
int main(void);
{
T x[2];
uintptr_t p = (uintptr_t)&x[0];
uintptr_t q = (uintptr_t)&x[1];
assert(q - p == sizeof(T)); // implementation-defined whether true
}
because the transformation performed by converting a pointer to an integer of the same size, or vice versa, is implementation-defined. I think it's required to be bijective, but I could be wrong about that, and it is definitely not required to preserve any of the above invariants.
I am working on a debugger (in C) which deals with embedded system whose memory has an addressable size of 16-bits. This means that at address x you have a 16-bits value, at (x + 1) you have another 16-bits value, etc. When working with data that comes from the target memory, I therefore have to convert the lengths between target bytes lengths and host bytes lengths. If I want to read 10 (16-bit) bytes from the target, I therefore have to allocate a 20 (8-bit) bytes in the debugger, which runs on the host.
A big number of bugs come from the fact that I forget to do these conversion, and I do semantically invalid arithmetic operations, such as adding an host memory address (i.e. the buffer allocated on my host) to a length in target bytes lengths. This makes no sense, and the length should be converted to an equivalent host bytes length prior to the addition (effectively multiply the length by 2).
My question is: is there any way that I can get the compiler to protect me from my own mistakes?
Both length types are integers. Is there way, for example, through type magic, to tell the compiler that this kind of integer can't be added to this other type of integers, for example?
Thanks!
I was reading through a presentation on the implementation of malloc, and on slide 7 it suggests storing a regions size and availability in a single word to save space. The alternative is to use two words, which is wasteful as the availability bit only needs to be 0 or 1.
This is the given explanation:
If blocks are aligned, low-order address bits are always 0
Why store an always-0 bit?
Use it as allocated/free flag! When reading size word, must mask out this bit
http://courses.engr.illinois.edu/cs241/sp2012/lectures/09-malloc.pdf
But I'm not really understanding how this works and how it could be implemented in C. Why is one bit of the size integer always 0?
If blocks are aligned, low-order address bits are always 0
This is the key to understanding what it going on. Many CPUs require that multibyte primitive values be stored at addresses divisible by the number of bytes in the primitive: 16-bit primitives need to be stored at even addresses; 32-bit ints need to be stored at addresses divisible by four, and so on. An attempt to access an int through a pointer that corresponds to an odd address results in a bus error.
In systems like that malloc must always return an address suitable for storing any primitive supported by the given CPU. Therefore, if CPU supports 32-bit integers, all addresses returned by malloc must be divisible by 4. Such addresses are said to be aligned. To comply, malloc implementations pad sizes blocks requested by the program by 0 to 3 bytes at the end to have length divisible by 4. As a consequence of this decision, the last two bits of an address of an aligned block will always be zero. An implementation of malloc can use these bits for its own purposes, as long as they are "masked out" before returning the result to callers.
malloc(3) (as specified by Posix) should
return a fresh block of memory; or NULL on failure; the returned pointer is not an alias of any other pointer in the program
return a suitably aligned block of memory. Alignment constraints are compiler, ABI, and processor specific. (Often, the alignment should be two words).
The size is not always zero. (actually, it is never zero). You could round it up to a multiple of two words, and use the last bit as a used/free bit.
However, pointers returned by malloc should be suitably aligned, e.g. to 8 bytes. So their bottom 3 bits are zero, and the allocated size in bytes of the malloc-ed zone is a multiple of 8 bytes (above the requested size passed to malloc), so the last 3 bits are zero (and you could use the last bit for other purposes, e.g. a used/free bit).
Why is one bit of the size integer always 0?
I see why this is confusing, but I don't think that's what they're saying on slide 7. They're saying that the low-order address bits are always 0.
Memory addresses of objects are aligned to specific boundaries, which means objects are aligned to a memory address that is a multiple of their size.
So a 64-bit integer is aligned to an eight-byte boundary;
0x7fff315470d8
If a pointer is always aligned to an eight-byte boundary, then the low-order three bits are always zero. ie: 0x816 = 10002
Basically, you can stick whatever you want in those low-order bits, so long as you take them out before dereferencing the pointer. In the 64-bit case you have 3 bits that are always 0, so you can store 3 "flags". In the case of this power point, they're saying take that lowest bit and use it for an "allocated" flag. Stick a 1 in it as long as the memory is allocated, mask it out when you send the pointer to the user.
#include <stdio.h>
int main(void){
int *ptr;
printf("the value of ptr is %p",ptr);
}
This gives me 0x7fffbd8ce900, which is only 6 bytes. Should it be 8 bytes (64bit)?
Although a pointer is 64 bits, current processors actually only support 48 bits, so the upper two bytes of an address are always either 0000 or (due to sign-extension) FFFF.
In the future, if 48 bits is no longer enough, new processors can add support for 56-bit or 64-bit virtual addresses, and existing programs will be able to utilize the additional space since they're already using 64-bit pointers.
That just means the first two bytes are zero (which, incidentally, is currently guaranteed for x86-64 chips—but that doesn't mean anything in this case, since your pointer is not initialized). %p is allowed to truncate leading zeroes, just like any other numeric type. %016p, however, is not. This should work fine:
printf("the value of ptr is %016p", ptr);
Because the 6-bytes address is just the virtual address(offset of the actual physical address). In physical architecture(X86 for instance), memory is divided into portions that may be addressed by a single index register without changing a 16-bit segment selector. In real mode of X86-CPU, a segment is always using 16-bit(2-bytes) segment-selector, which will dynamically decided by the Operating-System at the very beginning when your program started to run(i.e. creating actual running process).
Hence, if your variable have the 48-bit address 0x7fffbd8ce900, and your program have the segment-selector offset 08af, and the real address of the variable is (0x08af<<48)+0x7fffbd8ce900 = 0x08af7fffbd8ce900, which is 64-bit.
further reading pls turn to:
x86 memory segmentation
X86-64, Linux, Windows.
Consider that I'd want to make some sort of "free launch for tag pointers". Basically I want to have two pointers that point to the same actual memory block but whose bits are different. (For example I want one bit to be used by GC collection or for some other reason).
intptr_t ptr = malloc()
intptr_t ptr2 = map(ptr | GC_FLAG_REACHABLE) //some magic call
int* p = int*(ptr);
int* p2 = int*(ptr2);
*p = 10;
*p2 = 20;
assert(*p == 20)
assert(p != p2)
On Linux, mmap() the same file twice. Same thing on Windows really, but it has its own set of functions for that.
Mapping the same memory (mmap on POSIX as Ignacio mentions, MapViewOfFile on Windows) to multiple virtual addresses may provide you some interesting coherency puzzles (are writes at one address visible when read at another address?). Or maybe not. I'm not sure what all the platform guarantees are.
More commonly, one simply reserves a few bits in the pointer and shifts things around as necessary.
If all your objects are aligned to 8-byte boundaries, it's common to simply store tags in the 3 least-significant bits of a pointer, and mask them off before dereferencing (as thkala mentions). If you choose a higher alignment, such as 16-bytes or 32-bytes, then there are 3 or 5 least-significant bits that can be used for tagging. Equivalently, choose a few most-significant bits for tagging, and shift them off before dereferencing. (Sometimes non-contiguous bits are used, for example when packing pointers into the signalling NaNs of IEEE-754 floats (223 values) or doubles (251 values).)
Continuing on the high end of the pointer, current implementations of x86-64 use at most 48 bits out of a 64-bit pointer (0x0000000000000000-0x00007fffffffffff + 0xffff800000000000-0xffffffffffffffff) and Linux and Windows only hand out addresses in the first range to userspace, leaving 17 most-significant bits that can be safely masked off. (This is neither portable nor guaranteed to remain true in the future, though.)
Another approach is to stop considering "pointers" and simply use indices into a larger memory array, as the JVM does with -XX:+UseCompressedOops. If you've allocated a 512MB pool and are storing 8-byte aligned objects, there are 226 possible object locations, so a 32-value has 6 bits to spare in addition to the index. A dereference will require adding the index times the alignment to the base address of the array, saved elsewhere (it's the same for every "pointer"). If you look at things carefully, this is simply a generalization of the previous technique (which always has base at 0, where things line up with real pointers).
Once upon a time I worked on a Prolog implementation that used the following technique to have spare bits in a pointer:
Allocate a memory area with a known alignment. malloc() usually allocates memory with a 4-byte or 8-byte alignment. If necessary, use posix_memalign() to get areas with a higher alignment size.
Since the resulting pointer is aligned to intervals of multiple bytes, but it represents byte-accurate addresses, you have a few spare bits that will by definition be zero in the memory area pointer. For example a 4-byte alignment gives you two spare bits on the LSB side of the pointer.
You OR (|) your flags with those bits and now have a tagged pointer.
As long as you take care to properly mask the pointer before using it for memory access, you should be perfectly fine.