I pass a pointer to a big array defined in Python (millions of rows) to a C-function with ctypes.
In the C-part I do pointer arithmetic on this array to access the values of it. Now I get some weird/false calculations and I think it's because of the pointer arithmetic.
As I understand pointers are always integers so the last entry of the array I can access would be the maximum integer value of that specific system.
So what does happen if I try to access *(array+i) where i is bigger than the maximum integer? And if that's the case how can I access remaining entrys of the array?
Most modern operating systems use a flat memory model, which means that a pointer is able to represent any address in the virtual address space.
On 64-bit systems, pointers are 64 bits, on 32-bit systems, they are 32 bits.
On 32-bit operating systems, the virtual address space of every process is 2^32 bytes, which is several billion bytes (i.e. several gigabytes).
On 64-bit operating systems, the virtual address space of every process is 2^64 bytes, which is an astronomically high number.
So what does happen if I try to access *(array+i) where i is bigger than the maximum [pointer]?
You will never encounter such a situation on systems with a flat memory model, since a pointer is always large enough to represent any address in a process' virtual address space.
It may be theoretically possible for you to encounter such a situation on a system with a segmented memory model. However, that is certainly not the reason why you are having the problem you describe.
As I understand pointers are always integers so the last entry of the array I can access would be the maximum integer value of that specific system.
The last element of the array that you can legally access depends on how much memory you allocated for the array.
Related
I just restart the C programming study. Now, I'm studying the memory storage capacity and the difference between bit and byte. I came across to this definition.
There is a calculation to a 32 bits system. I'm very confused, because in this calculation 2^32 = 4294967296 bytes and it means about 4 Gigabyte. My question is: Why 2 raised to 32 power results in a number in bytes instead of bits ?
Thanks for helping me.
Because the memory is byte-addressable (that is, each byte has its own address).
There are two ways to look at this:
A 32-bit integer can hold one of 2^32 different values. Thus, a uint32_t can represent the values from 0 to 4294967295.
A 32-bit address can represent 2^32 different addresses. And as Scott said, on a byte-addressable system, that means 2^32 different bytes can be addressed. Thus, a process with 32-bit pointers can address up to 4 GiB of virtual memory. Or, a microprocessor with a 32-bit address bus can address up to 4 GiB of RAM.
That description is really superficial and misses a lot of important considerations, especially as to how memory is defined and accessed.
Fundamentally an N-bit value has 2N possible states, so a 16-bit value has 65,536 possible states. Additionally, memory is accessed as bytes, or 8-bit values. This was not always the case, older machines had different "word" sizes, anywhere from 4 to 36 bits per word, occasionally more, but over time the 8-bit word, or "byte", became the dominant form.
In every case a memory "address" contains one "word" or, on more modern machines, "byte". Memory is measured in these units, like "kilowords" or "gigabytes", for reasons of simplicity even though the individual memory chips themselves are specified in terms of bits. For example, a 1 gigabyte memory module often has 8 gigabit chips on it. These chips are read at the same time, the resulting data combined to produce a single byte of memory.
By that article's wobbly definition this means a 16-bit CPU can only address 64KB of memory, which is wrong. DOS systems from the 1980s used two pointers to represent memory, a segment and an offset, and could address 16MB using an effective 24-bit pointer. This isn't the only way in which the raw pointer size and total addressable memory can differ.
Some 32-bit systems also had an alternate 36-bit memory model that allowed addressing up to 64GB of memory, though an individual process was limited to a 4GB slice of the available memory.
In other words, for systems with a singular pointer to a memory address and where the smallest memory unit is a byte then the maximum addressable memory is 2N bytes.
Thankfully, since 64-bit systems are now commonplace and a computer with > 64GB of memory is not even exotic or unusual, addressing systems are a lot simpler now then when having to work around pointer-size limitations.
We say that memory is byte-addressable, you can think like byte is the smallest unit of memory so you are not reading by bits but bytes. The reason might be that the smallest data type is 1 byte, even boolean type in c/c++ is 1 byte.
What is meant by the size of a pointer? Shouldn't the size of pointer depend on the type? Most of the sources say the size of a pointer if 4 or 8 bytes. I need some clarity on this claim.
For size of a pointer I would mean the number of bits (or bytes) necessary to hold that pointer in memory, or send its value across some channel, and this is (possibly) different from the size of the object the pointer points to.
Then, it can be assumed as fairly true the affirmation that pointer sizes are commonly 32 or 64 bits (4 or 8 bytes), in the sense that the systems much talked about (computers, smartphones and tablets) have pointers of that size.
But there are other systems around, smaller like DOS-based PCs or microcontrollers for embedded systems, where a pointer can be 16 bits wide or even less, and bigger systems with bus width of, say, 128 bits.
I worked in the past with the Intel 8051 CPU, which had pointers 8 bits wide, 16 bits wide, and 24 bits wide. Of course they were not freely mixable... That CPU was indeed quite strange, having about 3-4 different (and little) areas of memory; a "specialized" pointer could point only in its special area, while the 24 bit wide one could point to any area because in the upper byte there was a "selector".
Another matter is the size of the object the pointer points to. On normal computers it is a byte, but sometimes, on certain systems, it is impossible to address bytes on odd addresses in this way, so pointer arithmetic gets complicated. The 8051 (I like it!) had even pointers pointing to bits! So the size of the pointed object was actually an eight of byte, and incrementing the pointer by one could, or could not, address a different memory location than before.
Data is stored in memory. That memory has an address. Pointers hold the memory address for where the data starts.
Specifically, pointers usually hold the address of the "first byte" of data where the type resides (note that technically, the first byte might contain the last bits of data, depending on endianness).
i.e., if a long double is 128bit (16 bytes), the pointer value will point to the first byte and the pointer type will indicate the numbers of bytes that should be read.
Should you "cast" the long double pointer in the example to an int * (an int pointer), only sizeof(int) bytes would be read - but the value, the address of the first byte, will remain the same.
Hence, the pointer value is oblivious to the size of the data, the pointer only needs to be large enough to contain the address of the first byte. For this reason, usually pointers have the same length which is derived from a computer's "address space".
It is very similar to a catalog card in a library. Just like a "book address" in a library depends on the size of the library, the pointer value (the memory address) depends on the size of the computer's "address space", not the size of the type.
On most 32 bit and 64 bit CPUs, the address space is limited to either 32 or 64 bits. However, some systems have special address spaces for special pointers (such as function pointers)... this is mostly obsolete. It was more in use when CPUS were smaller than 32 bits and the "address space" was limited.
Note that values in the address space (pointers) can point to any location on the hardware (usually a byte in memory, but sometimes a register or a piece of hardware)... this is why the OS (kernel), leveraging some hardware support, will usually expose a "virtual" address space per process, shielding the hardware and other processed from a misbehaving process.
P.S.
I loved the answer given by #linuxfansaysReinstateMonica ... However, I found that I wanted to clarify some of the information in that answer. You should really read it. This answer is mostly a clarification for their answer.
I'm programmer-beginner, but I want to understand the things a bit more deeply. I did some research and read quite a lot of text, but I'm still yet to understand some things..
When coding a basic thing (in C):
int myNumber;
myNumber = 3;
printf("Here's my number: %d", myNumber);
I found out that (mainly on 32-bit CPU) integer takes place of 32 bits = 4 bytes. So at first line of my code CPU goes into the memory. The memory is byte-addressable, so CPU chooses 4 continuous bytes for my variable and stores the address to first (or last) byte.
On the second line of my code CPU uses his stored address of the MyNumber variable, goes to that address in the memory and finds there 32 bits of reserved space. His task now is to store there the number "3", so he fills those four bytes with the sequence 00000000-00000000-00000000-00000011.
On the third line it does the same - CPU goes to that address in memory and loads the number stored in that address.
(First question - Do I understand it right?)
What I don't understand is this:
The size of that address (pointer to that variable) is 4bytes in 32-bit CPU. (Thats why 32-bit CPU can use max 4GB of memory - because there are only 2^32 different addresses of binary length 32)
Now, where the CPU stores these addresses? Does he have some sort of its own memory or cache to store that? And why it stores the 32 bit long address to 32 bit long integer? Wouldn't it be better to simply store in its cache that actual number than the pointer to that when the sizes are the same?
And last one - if it stores somewhere in its own cache the addresses to all those integers and the lenghts are the same (4 bytes), it will need exactly the same space for storing the addresses as for the actual variables. But variables can take up to 4GBs of space so CPU must have 4GB of its own space to store the addresses to those variables. And that sounds strange..
Thank you for help!
I'm trying to understand that but it's so tough.. :-[
(First question - Do I understand it right?)
The first thing to recognise is that the value might not be stored in main memory at all. The compiler might decide to store it in a register instead, as this is more optimal.1
The memory is byte-addressable, so CPU chooses 4 continuous bytes for my variable and stores the address to first (or last) byte.
Assuming that the compiler does decide to store it in main memory, then yes, on a 32-bit machine, an int is typically 4 bytes, so 4 bytes will be allocated for storage.
The size of that address (pointer to that variable) is 4bytes in 32-bit CPU. (Thats why 32-bit CPU can use max 4GB of memory - because there are only 2^32 different addresses of binary length 32)
Note that the width of an int and the width of a pointer don't have to be the same, so there's not necessarily a connection with the size of the address space.
Now, where the CPU stores these addresses?
In the case of local variables, the address is effectively hardcoded into the executable itself, typically as an offset from the stack pointer.
In the case of dynamically-allocated objects (i.e. stuff that's been malloc-ed), the programmer typically maintains a corresponding pointer variable (otherwise there would be a memory leak!). That pointer might also be dynamically-allocated (in the case of a complex data structure), but if you go back far enough, you'll eventually reach something that's a local variable. In which case, the above rule applies.
But variables can take up to 4GBs of space so CPU must have 4GB of its own space to store the addresses to those variables.
If your program consists of independently malloc-ing millions of ints, then yes, you'd end up with just as much storage required for the pointers. But most programs don't look like that. You typically allocate much bigger objects (like an array, or a big struct).
cache
The specifics of where stuff is stored is architecture-specific. On a modern x86, there's typically 2 or 3 layers of cache sitting between the CPU and main memory. But the cache is not independently addressable; the CPU cannot decide to store the int in cache instead of main memory. Rather, the cache is effectively a redundant copy of a subset of main memory.
Another thing to consider is that the compiler will typically deal with virtual addresses when allocating storage for objects. On a modern x86, these are mapped to physical addresses (i.e. addresses that correspond to physical storage bytes in main memory) by dedicated hardware, along with OS support.
1. Alternatively, the compiler may be able to optimise it away entirely.
On the second line of my code CPU uses his stored address of the MyNumber variable, goes to that address in the memory and finds there 32 bits of reserved space.
Nearly correct. Memory is basically unstructured. The CPU can't see that there are 32 bits of "reserved space". But the CPU was instructed to read 32 bits of data, so it reads 32 bits of data starting from the specified address. Then it just has to hope/assume that those 32 bits actually contain something meaningful.
Now, where the CPU stores these addresses? Does he have some sort of its own memory or cache to store that? And why it stores the 32 bit long address to 32 bit long integer? Wouldn't it be better to simply store in its cache that actual number than the pointer to that when the sizes are the same?
The CPU has a small number of registers, which it can use to store data (common CPUs have 8, 16 or 32 registers, so they can only hold the particular variables that you're working with here and now). So to answer the last part first, yes, the compiler certainly might (and probably will) generate code to just store your int into a register, instead of storing it in a memory, and telling the CPU to load it from a specified address.
As for the other part of the question: ultimately, every part of the program is stored in memory. Part of it is a stream of instructions, and part of it is in chunks of data scattered around memory.
There are a few tricks that help with locating the data the CPU needs: part of the program's memory contains the stack, which typically stores local variables while they're in scope. The CPU always maintains a pointer to the top of the stack in one of its registers, so it can easily locate data on the stack, simply by modifying the stack pointer with a fixed offset. Instructions can directly contain such offsets, so in order to read your int, the compiler could for example generate code which writes the int to the top of the stack when you enter the function, and then when you need to refer to that function, have code which reads the data found at the address the stack pointer points to, plus the small offset needed to locate your variable.
And also keep in mind that the adresses your program sees may not be (or rather rarely are) physical adresses starting from the 'beginning of the memory' or 0. Mostly they are offsets into a specifc memory block where the memory manager knows the real address and the accesses via base+offest as the real data storage.
And we do need the memory since caches are limited ;-)
Mario
Internal to the CPU there is one register that contains the address of the next instruction to be executed. The instructions themselves keep information where the variable is. If the variable is optimized, the instruction may point to a register, but in general, the instruction will have an address of the variable being accessed. Your code, after compiled and loaded in memory, have all that embedded! I recommend looking into assembly language to get a better understanding of all that. Good luck!
I am reading some C text at the address:
https://cs.senecac.on.ca/~lczegel/BTP100/pages/content/compu.html
In the section: Addressible Memory they say that "The maximum size of addressable primary memory depends upon the size of the address registers."
I do not understand why is that.
Can anyone give me a clear explanation, please?
Thanks a lot.
If you have 32-bit registers, then the highest address you can store in a single register is 2^32-1, so you can address 2^32 units (in modern computers, units are almost always bytes). A larger number simply won't fit.
You can get around this by using memory addresses that are larger than a single register can hold (and some CPUs/operating systems have features for doing so), but using addresses/pointers will be slower because it has to fiddle with multiple registers.
As an example, suppose you have 32-bit registers but 64-bit pointers and want to increment a pointer to find the next item in an array of char (++p). Instead of performing a simple increment instruction, the processor will have to
Increment the lower 32 bits;
check if the result is zero (overflow);
increment the upper half as well if overflow occurred.
Simplifying a bit, this means it has to perform a branch (if-then-else) instruction, which is one of the slowest and most complex instructions a modern CPU performs.
(See, e.g., x86 memory segmentation on the Wikipedia for a multi-register addressing scheme used in Intel processors.)
Keeping it simple: the address registers are used to store and refer to addresses of memory; since their size and number is fixed, there is a maximum address.
Obviously you can't exploit more memory than what is addressable (because the machine wouldn't know how to refer to it), so the usable memory is in fact limited by the maximum address that can be expressed by the address registers.
If you have 1 address register, holding a 16 bit address, you can have a maximum of 2^16 - 1 addresses.
However many registers, the number of addresses they can point to will be limited by their width (number of bits).
Thus, the maximum size of addressable primary memory depends upon the size of the address registers.
Can we anyhow change the size of the pointer from 2 bytes so it can occupy more than 2 bytes?
Sure, compile for a 32 (or 64) bit platform :-)
The size of pointers is platform specific, it would be 2 bytes only on 16-bit platforms (which have not been widely used for more than a decade - nowadays all mainstream [update](desktop / laptop / server)[/update] platforms are at least 32 bits).
If your pointer size is 2 byte that means you're running on a 16-bit system.
The only way to increase the pointer size is to use a 32-bit or 64-bit system instead (which would mean any desktop or laptop computer built in the last 15 years or so).
If you're running on some embedded device that uses 16-bit, your only option would be to switch to another device which uses 32-bits (or just live with your pointers being 16-bit).
When a processor is said to be "X-bit" (where X is 16, 32, 64, etc), that X refers to the size of the memory address register. Thus a 16-bit system has a memory address register of 2 bytes.
You cannot cast a 4-byte address to anything smaller because it would lose part of where it's pointing to. (A 2-byte memory address register can only point to 2^16=64KB of memory, whereas a 4-byte register can point to 2^32=4GB of memory.)
You can always "step-up" (ie, run a 32-bit software application on a 64-bit computer) because there's no loss in pointer range. But you can never step down, which is why 64-bit programs don't run on 32-bit systems.
Think of a pointer as a number, only instead of an actual value used for computation, it's the number of a 'slot' in the memory map of the system.
A pointer must be able to represent the highest position of the memory map. That is, it must have at least the amount of bytes required to represent the number of the highest position.
In a 16-bit system, the highest possible position is 0xFFFF (a 16-bit number with all the bits set to 1). A pointer must also have 16 bits, so it can reach that number.
Generalizing, in an X-bit system, a pointer will have X bits.
You can store a pointer in a larger variable, the same way you can store the number 1 in a char, in an int, or an unsigned long long if you wanted to; but there's little point to that: think that, the same way a shorter pointer won't be able to reach the highest memory position, a longer pointer would be able to point to things that can't actually exist in memory, so why have it?
Also, you'd have to 'trick' the compiler for that. If you use the pointer notation in your code, the compiler will always use the correct amount of bytes for it. You can instruct the compiler to compile for another platform, though.