I know that each two letters in a hexdecimal address represents a byte, meaning that 0xFFFF is a 16bit address and can represent 65,536 bytes of memory, if the system is byte-addressable. However, if talking about a number of bits that is not a multiple of 8 (such as 14bit address), how can the operating system represnt these addresses?
0xFFFF -> 16 bit (e.g. Virtual memory address)
14 bit -> 0xFFF ? (Physical memory address)
One might say that the system has to be not byte addressable to access a not multiple of 8 address.. Then what will it be? I want the 16 bit of the virtual address to be byte addressable so we can easily access the data stored at that address, and I want to represent it in C code, but I have trouble representing the 14 bit physical memory addresses.
#define MAX_VIRT_ADDR ((0xffff) - 1) /* 65,536 */
#define MAX_PHYS_ADDR ((?) - 1) /* Max of 14bit physical memory space */
The addresses are still just numbers. On a system with a 14-bit address bus, they go from 00000000000000 (binary) up to 11111111111111 (binary).
We can also write those numbers in decimal: they go from 0 up to 16383.
Or we can write them in hexadecimal: they go from 0 up to 3FFF (or 0000 up to 3FFF).
Or in octal: they go from 0 up to 37777 (or 00000 up to 37777).
Or in any other system we like.
Typically a system based on 8-bit bytes will allow 2 bytes to be used to access a memory address. If the system has some kind of memory protection unit, and it's configured appropriately, then addresses above 3FFF may cause some kind of invalid address exception (i.e. a segfault). Otherwise, usually the extra bits are just ignored, so that no matter whether the program accesses "address" 0x0005, 0x4005, 0x8005, or 0xC005, the actual address sent to the address bus is 5 (binary: 00000000000101).
Your maximum "virtual" 16-bit address is 0xFFFF. Your maximum "physical" 14-bit address is 0x3FFF.
There's no rule that says address sizes have to be powers of 2, or that they have to be the same size as the words being addressed. It's massively more convenient to do things that way, but not required.
The old Motorola 68K had 32-bit words but a 24-bit address bus - address values were stored in 32-bit words with the upper 8 bits left unused.
As for mapping your 16-bit "virtual" address space onto a 14-bit "physical" address space, treat the upper two bits in the virtual address as a page number, treat the lower 14 bits as the offset into the page, map them directly to "physical" addresses. Store in a 16-bit type like uint16_t, then use macros to extract page number and address like so:
#define PAGENO(vaddr) ((0xC000 & vaddr) >> 14)
#define PHADDR(vaddr) (0x3FFF & vaddr)
It's simple: your 14bit address is
struct {
unsigned addr : 14;
};
You simply ignore 2 bits of the 16bit value.
Another way is to sign extend the 14bit value to 16bit. That's what 64bit systems like AMD64 or ARM64 do. They have 43-56 bits address space depending on the CPU and that is sign extended to 64bit. Addresses where the top bits aren't all 0 or all 1 are illegal.
Exotic systems where bytes have more than 8 bits still (as far as I know) use the same addressing on byte level. Only the size of a byte changes. Certain digital signal processors like this do exist in the real world, although those would not run on an OS but get coded as "bare metal". Traditionally, DSP programming is also most often done in assembler rather than C.
So for any system with 16 bit wide address bus, you'll have:
#define MAX_VIRT_ADDR 0xffffu /* 65,535 */
#define MAX_PHYS_ADDR 0xffffu /* 65,535 */
And the size of one byte is irrelevant. If you'd design the system in any other way, you'd probably lose the advantage of having a larger byte size.
Related
I read that a 64-bit machine actually uses only 48 bits of address (specifically, I'm using Intel core i7).
I would expect that the extra 16 bits (bits 48-63) are irrelevant for the address, and would be ignored. But when I try to access such an address I got a signal EXC_BAD_ACCESS.
My code is:
int *p1 = &val;
int *p2 = (int *)((long)p1 | 1ll<<48);//set bit 48, which should be irrelevant
int v = *p2; //Here I receive a signal EXC_BAD_ACCESS.
Why this is so? Is there a way to use these 16 bits?
This could be used to build more cache-friendly linked list. Instead of using 8 bytes for next ptr, and 8 bytes for key (due to alignment restriction), the key could be embedded into the pointer.
The high order bits are reserved in case the address bus would be increased in the future, so you can't use it simply like that
The AMD64 architecture defines a 64-bit virtual address format, of which the low-order 48 bits are used in current implementations (...) The architecture definition allows this limit to be raised in future implementations to the full 64 bits, extending the virtual address space to 16 EB (264 bytes). This is compared to just 4 GB (232 bytes) for the x86.
http://en.wikipedia.org/wiki/X86-64#Architectural_features
More importantly, according to the same article [Emphasis mine]:
... in the first implementations of the architecture, only the least significant 48 bits of a virtual address would actually be used in address translation (page table lookup). Further, bits 48 through 63 of any virtual address must be copies of bit 47 (in a manner akin to sign extension), or the processor will raise an exception. Addresses complying with this rule are referred to as "canonical form."
As the CPU will check the high bits even if they're unused, they're not really "irrelevant". You need to make sure that the address is canonical before using the pointer. Some other 64-bit architectures like ARM64 have the option to ignore the high bits, therefore you can store data in pointers much more easily.
That said, in x86_64 you're still free to use the high 16 bits if needed (if the virtual address is not wider than 48 bits, see below), but you have to check and fix the pointer value by sign-extending it before dereferencing.
Note that casting the pointer value to long is not the correct way to do because long is not guaranteed to be wide enough to store pointers. You need to use uintptr_t or intptr_t.
int *p1 = &val; // original pointer
uint8_t data = ...;
const uintptr_t MASK = ~(1ULL << 48);
// === Store data into the pointer ===
// Note: To be on the safe side and future-proof (because future implementations
// can increase the number of significant bits in the pointer), we should
// store values from the most significant bits down to the lower ones
int *p2 = (int *)(((uintptr_t)p1 & MASK) | (data << 56));
// === Get the data stored in the pointer ===
data = (uintptr_t)p2 >> 56;
// === Deference the pointer ===
// Sign extend first to make the pointer canonical
// Note: Technically this is implementation defined. You may want a more
// standard-compliant way to sign-extend the value
intptr_t p3 = ((intptr_t)p2 << 16) >> 16;
val = *(int*)p3;
WebKit's JavaScriptCore and Mozilla's SpiderMonkey engine as well as LuaJIT use this in the nan-boxing technique. If the value is NaN, the low 48-bits will store the pointer to the object with the high 16 bits serve as tag bits, otherwise it's a double value.
Previously Linux also uses the 63rd bit of the GS base address to indicate whether the value was written by the kernel
In reality you can usually use the 48th bit, too. Because most modern 64-bit OSes split kernel and user space in half, so bit 47 is always zero and you have 17 top bits free for use
You can also use the lower bits to store data. It's called a tagged pointer. If int is 4-byte aligned then the 2 low bits are always 0 and you can use them like in 32-bit architectures. For 64-bit values you can use the 3 low bits because they're already 8-byte aligned. Again you also need to clear those bits before dereferencing.
int *p1 = &val; // the pointer we want to store the value into
int tag = 1;
const uintptr_t MASK = ~0x03ULL;
// === Store the tag ===
int *p2 = (int *)(((uintptr_t)p1 & MASK) | tag);
// === Get the tag ===
tag = (uintptr_t)p2 & 0x03;
// === Get the referenced data ===
// Clear the 2 tag bits before using the pointer
intptr_t p3 = (uintptr_t)p2 & MASK;
val = *(int*)p3;
One famous user of this is the V8 engine with SMI (small integer) optimization. The lowest bit in the address will serve as a tag for type:
if it's 1, the value is a pointer to the real data (objects, floats or bigger integers). The next higher bit (w) indicates that the pointer is weak or strong. Just clear the tag bits and dereference it
if it's 0, it's a small integer. In 32-bit V8 or 64-bit V8 with pointer compression it's a 31-bit int, do a signed right shift by 1 to restore the value; in 64-bit V8 without pointer compression it's a 32-bit int in the upper half
32-bit V8
|----- 32 bits -----|
Pointer: |_____address_____w1|
Smi: |___int31_value____0|
64-bit V8
|----- 32 bits -----|----- 32 bits -----|
Pointer: |________________address______________w1|
Smi: |____int32_value____|0000000000000000000|
https://v8.dev/blog/pointer-compression
So as commented below, Intel has published PML5 which provides a 57-bit virtual address space, if you're on such a system you can only use 7 high bits
You can still use some work around to get more free bits though. First you can try to use a 32-bit pointer in 64-bit OSes. In Linux if x32abi is allowed then pointers are only 32-bit long. In Windows just clear the /LARGEADDRESSAWARE flag and pointers now have only 32 significant bits and you can use the upper 32 bits for your purpose. See How to detect X32 on Windows?. Another way is to use some pointer compression tricks: How does the compressed pointer implementation in V8 differ from JVM's compressed Oops?
You can further get more bits by requesting the OS to allocate memory only in the low region. For example if you can ensure that your application never uses more than 64MB of memory then you need only a 26-bit address. And if all the allocations are 32-byte aligned then you have 5 more bits to use, which means you can store 64 - 21 = 43 bits of information in the pointer!
I guess ZGC is one example of this. It uses only 42 bits for addressing which allows for 242 bytes = 4 × 240 bytes = 4 TB
ZGC therefore just reserves 16TB of address space (but not actually uses all of this memory) starting at address 4TB.
A first look into ZGC
It uses the bits in the pointer like this:
6 4 4 4 4 4 0
3 7 6 5 2 1 0
+-------------------+-+----+-----------------------------------------------+
|00000000 00000000 0|0|1111|11 11111111 11111111 11111111 11111111 11111111|
+-------------------+-+----+-----------------------------------------------+
| | | |
| | | * 41-0 Object Offset (42-bits, 4TB address space)
| | |
| | * 45-42 Metadata Bits (4-bits) 0001 = Marked0
| | 0010 = Marked1
| | 0100 = Remapped
| | 1000 = Finalizable
| |
| * 46-46 Unused (1-bit, always zero)
|
* 63-47 Fixed (17-bits, always zero)
For more information on how to do that see
Allocating Memory Within A 2GB Range
How can I ensure that the virtual memory address allocated by VirtualAlloc is between 2-4GB
Allocate at low memory address
How to malloc in address range > 4 GiB
Custom heap/memory allocation ranges
Side note: Using linked list for cases with tiny key values compared to the pointers is a huge memory waste, and it's also slower due to bad cache locality. In fact you shouldn't use linked list in most real life problems
Bjarne Stroustrup says we must avoid linked lists
Why you should never, ever, EVER use linked-list in your code again
Number crunching: Why you should never, ever, EVER use linked-list in your code again
Bjarne Stroustrup: Why you should avoid Linked Lists
Are lists evil?—Bjarne Stroustrup
A standards-compliant way to canonicalize AMD/Intel x64 pointers (based on the current documentation of canonical pointers and 48-bit addressing) is
int *p2 = (int *)(((uintptr_t)p1 & ((1ull << 48) - 1)) |
~(((uintptr_t)p1 & (1ull << 47)) - 1));
This first clears the upper 16 bits of the pointer. Then, if bit 47 is 1, this sets bits 47 through 63, but if bit 47 is 0, this does a logical OR with the value 0 (no change).
I guess no-one mentioned possible use of bit fields ( https://en.cppreference.com/w/cpp/language/bit_field ) in this context, e.g.
template<typename T>
struct My64Ptr
{
signed long long ptr : 48; // as per phuclv's comment, we need the type to be signed to be sign extended
unsigned long long ch : 8; // ...and, what's more, as Peter Cordes pointed out, it's better to mark signedness of bit field explicitly (before C++14)
unsigned long long b1 : 1; // Additionally, as Peter found out, types can differ by sign and it doesn't mean the beginning of another bit field (MSVC is particularly strict about it: other type == new bit field)
unsigned long long b2 : 1;
unsigned long long b3 : 1;
unsigned long long still5bitsLeft : 5;
inline My64Ptr(T* ptr) : ptr((long long) ptr)
{
}
inline operator T*()
{
return (T*) ptr;
}
inline T* operator->()
{
return (T*)ptr;
}
};
My64Ptr<const char> ptr ("abcdefg");
ptr.ch = 'Z';
ptr.b1 = true;
ptr.still5bitsLeft = 23;
std::cout << ptr << ", char=" << char(ptr.ch) << ", byte1=" << ptr.b1 <<
", 5bitsLeft=" << ptr.still5bitsLeft << " ...BTW: sizeof(ptr)=" << sizeof(ptr);
// The output is: abcdefg, char=Z, byte1=1, 5bitsLeft=23 ...BTW: sizeof(ptr)=8
// With all signed long long fields, the output would be: abcdefg, char=Z, byte1=-1, 5bitsLeft=-9 ...BTW: sizeof(ptr)=8
I think it may be quite a convenient way to try to make use of these 16 bits, if we really want to save some memory. All the bitwise (& and |) operations and cast to full 64-bit pointer are done by compiler (though, of course, executed in run time).
According to the Intel Manuals (volume 1, section 3.3.7.1) linear addresses has to be in the canonical form. This means that indeed only 48 bits are used and the extra 16 bits are sign extended. Moreover, the implementation is required to check whether an address is in that form and if it is not generate an exception. That's why there is no way to use those additional 16 bits.
The reason why it is done in such way is quite simple. Currently 48-bit virtual address space is more than enough (and because of the CPU production cost there is no point in making it larger) but undoubtedly in the future the additional bits will be needed. If applications/kernels were to use them for their own purposes compatibility problems will arise and that's what CPU vendors want to avoid.
Physical memory is 48 bit addressed. That's enough to address a lot of RAM. However between your program running on the CPU core and the RAM is the memory management unit, part of the CPU. Your program is addressing virtual memory, and the MMU is responsible for translating between virtual addresses and physical addresses. The virtual addresses are 64 bit.
The value of a virtual address tells you nothing about the corresponding physical address. Indeed, because of how virtual memory systems work there's no guarantee that the corresponding physical address will be the same moment to moment. And if you get creative with mmap() you can make two or more virtual addresses point at the same physical address (wherever that happens to be). If you then write to any of those virtual addresses you're actually writing to just one physical address (wherever that happens to be). This sort of trick is quite useful in signal processing.
Thus when you tamper with the 48th bit of your pointer (which is pointing at a virtual address) the MMU can't find that new address in the table of memory allocated to your program by the OS (or by yourself using malloc()). It raises an interrupt in protest, the OS catches that and terminates your program with the signal you mention.
If you want to know more I suggest you Google "modern computer architecture" and do some reading about the hardware that underpins your program.
My question has two parts.
First, as a newbie to this address space, I would like to know what is the meaning of memory alignment of an address. I Googled about it but wanted to ask this question here as well since I found answers here very useful.
The second part of my question is related to alignment and programming: how do I find if an address is 4 byte aligned or not ?
Somewhere I read:
if(address & 0x3) // for 32 bit register
But I don't really know how this checks for a 4 byte alignment.
Could anyone explain it in detail?
Edit: It would be great If someone can draw pictorial view on this subject.
Thanks
Sequential addresses refer to sequential bytes in memory.
An address that is "4-byte aligned" is a multiple of 4 bytes. In other words, the binary representation of the address ends in two zeros (00), since in binary, it's a multiple of the binary value of 4 (100b). The test for 4-byte aligned address is, therefore:
if ( (address & 0x3) == 0 )
{
// The address is 4-byte aligned here
}
or simply
if ( !(address & 0x3) )
{
// The address is 4-byte aligned here
}
The 0x3 is binary 11, or a mask of the lowest two bits of the address.
Alignment is important since some CPU operations are faster if the address of a data item is aligned. This is because CPUs are 32-bit or 64-bit word based. Small amounts of data (say 4 bytes, for example) fit nicely in a 32-bit word if it is 4-byte aligned. If it is not aligned, it can cross a 32-bit boundary and require additional memory fetches. Modern CPUs have other optimizations as well that improve performance for address aligned data.
Here's a sample article regarding the topic of alignment and speed.
Here are some some nice diagrams of alignment.
#include <stdio.h>
int main(void){
int *ptr;
printf("the value of ptr is %p",ptr);
}
This gives me 0x7fffbd8ce900, which is only 6 bytes. Should it be 8 bytes (64bit)?
Although a pointer is 64 bits, current processors actually only support 48 bits, so the upper two bytes of an address are always either 0000 or (due to sign-extension) FFFF.
In the future, if 48 bits is no longer enough, new processors can add support for 56-bit or 64-bit virtual addresses, and existing programs will be able to utilize the additional space since they're already using 64-bit pointers.
That just means the first two bytes are zero (which, incidentally, is currently guaranteed for x86-64 chips—but that doesn't mean anything in this case, since your pointer is not initialized). %p is allowed to truncate leading zeroes, just like any other numeric type. %016p, however, is not. This should work fine:
printf("the value of ptr is %016p", ptr);
Because the 6-bytes address is just the virtual address(offset of the actual physical address). In physical architecture(X86 for instance), memory is divided into portions that may be addressed by a single index register without changing a 16-bit segment selector. In real mode of X86-CPU, a segment is always using 16-bit(2-bytes) segment-selector, which will dynamically decided by the Operating-System at the very beginning when your program started to run(i.e. creating actual running process).
Hence, if your variable have the 48-bit address 0x7fffbd8ce900, and your program have the segment-selector offset 08af, and the real address of the variable is (0x08af<<48)+0x7fffbd8ce900 = 0x08af7fffbd8ce900, which is 64-bit.
further reading pls turn to:
x86 memory segmentation
sizeof(long) returns 8 bytes but the &along (address of a long) is 12 hex digits (48-bits) or 6 bytes. On OS X 64-bit compiled with clang. Is there a discrepancy here or is this a genuine 64-bit address space?
In effect modern systems use 64 bit for addresses but not all of these are used by the system since 2 to the 64 is an amount of memory no one will use in a near future. Under linux you can find out what your machine actually uses by doing cat /proc/cpuinfo. On my machine I have e.g
address sizes : 36 bits physical, 48 bits virtual
so this would allow for 64 GiB of physical memory and 256 TiB of virtual memory. This is largely enough for what this machine will ever encounter.
I think you're conflating two different concepts. Sizeof tells you how long a long is, not now long an address is. A long takes up six bytes of memory, but the pointer to it (apparently) takes only 6 (or more likely, takes 8, but the first two are all-zero because of how the memory is laid out)
Let me expand some explanation about address spaces. While pointers are 8 bytes long in a 64 bit processor, the address space rarely needs that much. That allows 2^64 bytes to be addressed, much more than we need. So, for simplicity, many processors and compilers only use 48 of those bits. See this wikipedia link for a few useful diagrams:
http://en.wikipedia.org/wiki/X86-64#Canonical_form_addresses
An address us a memory location. On a 32-bit system it will be 32-bits long, on a 64-bit system 64-bits, and so on.
The sizeof of variable is how much memory it occupies. They are completely unrelated numbers.
sizeof(long) returns 8 bytes but the &along (address of a long) is 12 hex digits (48-bits) or 6 bytes.
Yes, because when you print a number the leading 0's aren't printed. For example:
int x = 0x0000000F;
printf( "%X", x );
// prints "F"
I'm not sure if this is the exact reason why you're seeing this, but current 64-bit processors typically don't actually have a 64-bit address space. The extra hardware involved would be a waste, since 48 bits of address space is more than I expect to need in my lifetime. It could be that you're seeing a truncated version of the address (i.e., the address is not being front-padded with zeros).
On 16 bit dos machine there is options like FP_SEG and FP_OFF for converting a pointer to linear address but since these method no more exist on 32 bit compiler what are other function that can do same on 32 bit machine??
They're luckily not needed as 32-bit mode is unsegmented and hence the addresses are always linear (a simplification, but let's keep it simple).
EDIT: The first version was confusing let's try again.
In 16-bit segmented mode (I'm exclusively referring to legacy DOS programs here, it'll probably be similar for other 16-bit x86 OSes) addresses are given in a 32-bit format consisting of a16-bit segment and a 16-bit offset. These are combined to form a 20-bit linear address (This is where the infamous 640K barrier comes from, 2**20 = 1MB and 384K are reserved for the system and bios leaving ~640K for user programs) by multiply the segment by 16 = 0x10 (equivalent to shifting left by 4) and adding the offset. I.e.: linear = segment*0x10 + offset.
This means that 2**12 segment:address type pointers will refer to the same linear address, so in general there is no way to obtain the 32-bit value used to form the linear address.
In old DOS programs that used far - segmented - pointers (as opposed to near pointers, which only contained an offset and implicitly used ds segment register) they were usually treated as 32-bit unsigned integer values where the 16 most significant bits were the segment and the 16 least significant bits the offset. This gives the following macro definitions for FP_SEG and FP_OFF (using the types from stdint.h):
#define FP_SEG(x) (uint16_t)((uint32_t)(x) >> 16) /* grab 16 most significant bits */
#define FP_OFF(x) (uint16_t)((uint32_t)(x)) /* grab 16 least significant bits */
To convert a 20-bit linear address to a segmented address you have many options (2**12). One way could be:
#define LIN_SEG(x) (uint16_t)(((uint32_t)(x)&0xf0000)>>4)
#define LIN_OFF(x) (uint16_t)((uint32_t)(x))
Finally a quick example of how it all works together:
Segmented address: a = 0xA000:0x0123
As 32-bit far pointer b = 0xA0000123
20-bit linear address: c = 0xA0123
FP_SEG(b) == 0xA000
FP_OFF(b) == 0x0123
LIN_SEG(c) = 0xA000
LIN_OFF(c) = 0x0123