I was studying musl libc malloc implementation and I am having hard time understanding the adjust_size function.
static int adjust_size(size_t *n)
{
/* Result of pointer difference must fit in ptrdiff_t. */
if (*n-1 > PTRDIFF_MAX - SIZE_ALIGN - PAGE_SIZE) {
if (*n) {
errno = ENOMEM;
return -1;
} else {
*n = SIZE_ALIGN;
return 0;
}
}
*n = (*n + OVERHEAD + SIZE_ALIGN - 1) & SIZE_MASK;
return 0;
}
For example in the first comparison, why they are not just comparing against PTRDIFF_MAX. It is what seems to be the intent from the comment above anyway, and why are they subtracting 1 from *n, I think that (*n-1) was being compared as unsigned instead of signed, so they are handling the case where *n is 0. But I do not know why this is being compared as unsigned in that case as it seems both positions would evaluate to signed numbers at the end.
Also why does they set *n to SIZE_ALIGN if it is 0? My understanding is that malloc should return NULL or a pointer where it can be passed to free without causing an issue if size is 0.
why they are not just comparing against PTRDIFF_MAX
Most malloc implementation allocate large chunks separately using mmap. Because mmap allocates memory in pages, n needs to be aligned to a page boundary (PAGE_SIZE), plus should include chunk header (which is aligned by SIZE_ALIGN).
This is why comparison is performed against PTRDIFF_MAX - SIZE_ALIGN - PAGE_SIZE instead of PTRDIFF_MAX - to make sure all possible future alignment adjustments won't cause chunk size to be greater than PTRDIFF_MAX.
why are they subtracting 1 from *n
Because n might be aligned later like this:
n = (n + SIZE_ALIGN + PAGE_SIZE - 1) & -PAGE_SIZE;
And resulting value should be less or equal to PTRDIFF_MAX. Value PTRDIFF_MAX - SIZE_ALIGN - PAGE_SIZE + 1 is still okay, so 1 is subtracted.
Also why does they set *n to SIZE_ALIGN if it is 0
Because adjusted chunk size should be greater or equal to SIZE_ALIGN bytes to fit OVERHEAD bytes of heap overhead plus requested data area should be able to fit 2 pointers used later from free. This alignment is assumed later in code.
I think that (*n-1) was being compared as unsigned instead of
signed, so they are handling the case where *n is 0. But I do not
know why this is being compared as unsigned in that case as it seems
both positions would evaluate to signed numbers at the end.
I think it could be written simpler (althought this might be incorrect, I am probably need to have a sleep):
static int adjust_size(size_t *n)
{
if (*n > PTRDIFF_MAX - SIZE_ALIGN - PAGE_SIZE + 1) {
errno = ENOMEM;
return -1;
}
*n = (*n + OVERHEAD + SIZE_ALIGN - 1) & SIZE_MASK;
return 0;
}
Related
I saw this interview question and wanted to know if my function is doing what it's supposed to or if there's a better way to do this.
Here's the exact quote of the question:
The operating system typically allocates memory in pages such that the base address of the page are 0, 4K, 8K etc. Given two addresses (pointers), write a function to find if two pointers are on the same page. Here's the function prototype: int AreOnSamePage (void * a, void * b);
Here's my implementation. I made it return 4 if it's between 4k and 8k. It returns 1 if it's between 0 and 4k and it returns -1 if it's over 8k away. Am I getting the right addresses? The interview question is worded vaguely. Is it correct to use long's since the addresses could be pretty big?
int AreOnSamePage(void* a, void* b){
long difference = abs(&a - &b);
printf("%ld %ld\n",(long)&a,(long)&b);
if(difference > 8000)
return -1;
if(difference >= 4000)
return 4;
return 1;
}
a and b are pointers, so the distance between them is:
ptrdiff_t difference = (ptrdiff_t) abs((char *)a - (char *) b)
But you don't need it.
Two pointers are on the same page, if
(uintptr_t)a / 4096 == ( uintptr_t ) b / 4096
Else they are on different pages.
So:
int AreOnSamePage(void* a, void* b) {
const size_t page_size = 4096;
if ( (uintptr_t) a / page_size == (uintptr_t) b / page_size)
return 1;
else
return 0;
}
There are many problems with your code.
You are comparing addresses of function parameters (they are side by side, on stack), not pointers
You for no reason compare the difference with 8000
4K != 4000
Imagine one address is 3K, other is 5K, according to your code, they are on the same page.
Bad choice of return values
The name AreOnSamePage() implies that the function returns either 0 or 1; I'd find it odd to have it return -1, 4 or other values.
If a page is 4KB, then it means you need 12 bits to index each byte inside a page (because 2^12 = 4096), so as long as the N-12 most significant bits of both pointer values compare equal, then you know they are on the same page (where N is the size of a pointer).
So you can do this:
#include <stdint.h>
static const uintptr_t PAGE_SIZE = 4096;
static const uintptr_t PAGE_MASK = ~(PAGE_SIZE-1);
int AreOnSamePage(void *a, void *b) {
return (((uintptr_t) a) & PAGE_MASK) == (((uintptr_t) b) & PAGE_MASK);
}
PAGE_MASK is a bit mask that has all N-12 most significant bits set to 1 and the 12 least significant bits set to 0. By doing the bitwise AND with an address, we effectively clear the least significant 12 bits (the offset into the page), so we can compare only the other bits that matter.
Note that uintptr_t is guaranteed to be wide enough to store pointer values, unlike long.
As already stated, you should use uintptr_t to proces the pointers. Your code is, however, wrong, as you test the distance, not the page. Also, you foget that computers use powers of two. 8000 is none; that would be 8192. similar for 4000.
The fastest approach for the test would be:
#include <stdbool.h>
#include <stdint.h>
// this should better be found in a system header:
#define PAGESIZE 4096U
bool samePage(void *a, void *b)
{
return ((uintptr_t)a ^ (uintptr_t)b) < PAGESIZE;
}
or:
return !(((uintptr_t)a ^ (uintptr_t)b) / PAGESIZE);
Note the result of the division will be converted to bool. If this is used as an inline, it will just tested for zero/not zero.
The XOR will zero all bits which are equal. So if any higher order bits differ, they will be set after XOR, and make the result >= PAGESIZE. This saves you one division or masking.
This requires PAGESIZE to be a power of two, of course.
Your aptempt to solve the interview's question is wrong.
You should be comparing a and b. Not &a and &b.
But even then it would still be wrong.
Consider pointer a points to last position of page 0 and pointer b points to first position of page 1. And page 1 is the one after page 0.
Their difference is 1. But they are in different pages.
In order to correctly implement it you should consider that a page is 4Kib long. 4Kib = 2^12 = 4096. So all the bits of a pair of pointers save for the last 12 will be equal if they are in the same page.
#include<stdint.h>
int AreOnSamePage(void* a, void* b){
return ((intptr_t)a & ~(intptr_t)0xFFF) ==
((intptr_t)b & ~(intptr_t)0xFFF);
}
A more concise but equivalent implementation :
int AreOnSamePage(void* a, void* b){
return ((intptr_t)a)>>12 == ((intptr_t)b)>>12;
}
I've been given the following c code for alignment
struct s *p, *new_p
p = (struct s*) malloc(sizeof(struct s) + BOUND -1);
new_p = (struct s*) (((int) p+BOUND-1) & ~(BOUND -1);
where BOUND represents 32 bytes. A line of cache is 32 bytes like in Pentium II and III but I cannot figure out the way p and new_p get aligned. Are both aligned or only new_p?
Also, I have this code for a line of cache of 64 B for a set associative cache with 8 blocks in each set and a size of 32 Kb:
int *tempA, *tempB;
...
pA= (int *) malloc (sizeof(int)*N + 63);
tempA = (int *)(((int)pA+63)&~(63));
tempB = (int *)((((int)pA+63)&~(63))+4096+64)
Accompanied with this remark: there will be a penalty if you access more than 8 address with a separation of 4 Kb.
The whole doesn't make much sense to me. Any ideas of what's going on?
Why not use _Alignas() (since C11)?
Casting a pointer to int is an invitation to disaster (aka undefined behaviour). Just think about a 64 bit machine with 32 bit (standard for most x86). If you need arithmetics on pointers, use uintptr_t (I would not recommend using intptr_t, though). However, even here, arithmetic on the value is still undefined (but very likely safe for platforms with single, linear address space).
Standard note: do not cast void * as returned by malloc().
Update:
Ok, lets take the code above and give it a proper formating and typing:
#include <stdint.h>
// align to this boundary (must be power of two!)
#define ALIGN_BOUNDARY 64U
Do not use magic numbers in your code! 2 months later you will wonder what that means.
int *tempA, *tempB;
How are those used?
int *pA = malloc (sizeof(int) * N + ALIGN_BOUNDARY - 1);
uintptr_t adjA = ((uintptr_t)pA + (ALIGN_BOUNDARY - 1)) & ~((uintptr_t) (ALIGN_BOUNDARY - 1);
This just rounds up the address to the next aligned boundary (here: 64 bytes).
tempA = (int *)adjA;
tempB = (int *)(adjA + 4096 + 64)
Not sure what the later is good for, but with the malloc given, that will result in disaster due to accessing beyond the allocated block if used with the same indexes (0..N) as *pA.
In any way, I would be very, very careful with this code. Not only it apparently is badly written/documented, but it seems also to contain errors.
I am learning the Redis source code , and in the zmalloc.c,
size_t zmalloc_size(void *ptr) {
void *realptr = (char*)ptr-PREFIX_SIZE;
size_t size = *((size_t*)realptr);
/* Assume at least that all the allocations are padded at sizeof(long) by
* the underlying allocator. */
if (size&(sizeof(long)-1)) size += sizeof(long)-(size&(sizeof(long)-1));
return size+PREFIX_SIZE;
}
I am confused with
if (size&(sizeof(long)-1)) size += sizeof(long)-(size&(sizeof(long)-1));
what's the effect of it? Memory padding?Then why sizeof(long)?
Yes, it seems to be to include the memory padding with the assumption that all allocations are padded at the sizeof(long) (as said by the comment).
Pseudo-code example:
size = 6 // as an example
sizeof(long) == 4
size & (sizeof(long) - 1) == 6 & (4 - 1) == 6 & 3 == 2
size += 4 - 2
size == 8 // two bytes of padding included
I'm pretty fresh in C though so you should probably not take my word for it. I'm not sure why one can assume that the underlying allocator will align at the size of long, perhaps it's only a decent approximation that is sufficient for zmalloc_size's use-case.
Here is a short snippet of a function reading lines.
How is that possible that it compares bufsize with ((size_t)-1)/2 ?
I imagined comparing a variable to eg. int - that is just impossible; to INT_MAX on the contrary it is correct, I think.
So how can that code actually work and give no errors?
int c;
size_t bufsize = 0;
size_t size = 0;
while((c=fgetc(infile)) != EOF) {
if (size >= bufsize) {
if (bufsize == 0)
bufsize = 2;
else if (bufsize <= ((size_t)-1)/2)
bufsize = 2*size;
else {
free(line);
exit(3);
}
newbuf = realloc(line,bufsize);
if (!newbuf) {
free(line);
abort();
}
line = newbuf;
}
/* some other operations */
}
(size_t)-1
This is casting the value -1 to a size_t. (type)value is a cast in C.
Since size_t is an unsigned type, this is actually the maximum value that size_t can hold, so it's used to make sure that the buffer size can actually be safely doubled (hence the subsequent division by two).
The code relies on some assumptions about bits and then does a well known hack for finding the maximum size_t value (provided that size_t doesn't accommodate more bits than the register, a safe bet on many machines).
First it fills a register up with 1 bits, then it casts it into a size_t data type, so the comparison will work. As long as that register is larger in number of bits than the size_t data type, then the (if any) unused 1 bits will be truncated, and you will get the largest unsigned number that can fit in size_t bits.
After you have that, it divides by two to get half of that number, and does the comparison to see if it seems to be safe to increase size without going over the "maximum" size_t. but by then, it's dividing a size_t data type, and comparing two size_t data types (a type safe operation).
If you really wanted to remove this bit-wizardy (ok, it's not the worst example of bit wizardy I've seen). Consider that the following snippet
else if (bufsize <= ((size_t)-1)/2)
bufsize = 2*size;
could be replaced with
else if (bufsize <= (MAX_SIZE/2)
bufsize = 2*size;
and be type safe without casting and more readable.
(size_t)-1 casts -1 to the type size_t, which results in SIZE_MAX (a macro defined in stdint.h), the maximum value that the size_t type can hold.
So the comparison is checking whether bufsize is less than or equal to one half the maximum value that can be contained in a size_t
size_t isn't being interpreted as a value, it's being used to cast the value of negative one to the type size_t.
((size_t)-1)/2
is casting -1 to a size_t and then dividing by 2.
The size_t in ((size_t)-1)/2) is simply being used as a cast: casting -1 to size_t.
The trick here is that size_t is unsigned, so the cast (size_t) -1 will be converted to the maximum value of size_t, or SIZE_MAX. This is useful in the context of the loop. However, I'd prefer to see SIZE_MAX used directly rather than this trick.
Here is a piece of code from the lib/xreadlink.c file in GNU Coreutils..
/* Call readlink to get the symbolic link value of FILENAME.
+ SIZE is a hint as to how long the link is expected to be;
+ typically it is taken from st_size. It need not be correct.
Return a pointer to that NUL-terminated string in malloc'd storage.
If readlink fails, return NULL (caller may use errno to diagnose).
If malloc fails, or if the link value is longer than SSIZE_MAX :-),
give a diagnostic and exit. */
char * xreadlink (char const *filename)
{
/* The initial buffer size for the link value. A power of 2
detects arithmetic overflow earlier, but is not required. */
size_t buf_size = 128;
while (1)
{
char* buffer = xmalloc(buf_size);
ssize_t link_length = readlink(filename, buffer, buf_size);
if(link_length < 0)
{
/*handle failure of system call*/
}
if((size_t) link_length < buf_size)
{
buffer[link_length] = 0;
return buffer;
}
/*size not sufficient, allocate more*/
free (buffer);
buf_size *= 2;
/*Check whether increase is possible*/
if (SSIZE_MAX < buf_size || (SIZE_MAX / 2 < SSIZE_MAX && buf_size == 0))
xalloc_die ();
}
}
The code is understandable except I could not understand how the check for whether the link's size is too big works, that is the line:
if (SSIZE_MAX < buf_size || (SIZE_MAX / 2 < SSIZE_MAX && buf_size == 0))
Further, how can
(SIZE_MAX / 2 < SSIZE_MAX)
condition be true on any system???
SSIZE_MAX is the maximum value of the signed variety of size_t. For instance if size_t is only 16 bits (very unlikely these days), SIZE_MAX is 65535 while ssize_max is 32767. More likely it is 32 bits (giving 4294967295 and 2147483647 respectively), or even 64 bits (giving numbers too big to type here :-) ).
The basic problem to solve here is that readlink returns a signed value even though SIZE_MAX is an unsigned one ... so once buf_size exceeds SSIZE_MAX, it's impossible to read the link, as the large positive value will result in a negative return value.
As for the "furthermore" part: it quite likely can't, i.e., you're right. At least on any sane system, anyway. (It is theoretically possible to have, e.g., a 32-bit SIZE_MAX but a 33-bit signed integer so that SSIZE_MAX is also 4294967295. Presumably this code is written to guard against theoretically-possible, but never-actually-seen, systems.)