was reading Robert Love's book, chapter 5 on syscalls, and found this simple example a bit questionable:
asmlinkage long sys_silly_copy(unsigned long *src, unsigned long *dst, unsigned long len)
{
unsigned long buf;
if (copy_from_user(&buf, src, len))
return -EFAULT;
...
}
As we see 'buf' is object of type 'unsigned long' and defined on the kernel stack, i.e. its initial value is likely garbage. Anyway is it valid to copy 'len' bytes in the stack where buf is, i.e. it could overwrite something useful? Perhaps this is fine only for this particular example?
It is very questionable. In fact, it's downright dangerous. I'll give the author the benefit of the doubt here since they're just trying to show how copy_from_user and copy_to_user work but they really should have provided an example that wasn't so dangerous.
Especially since the book waxes lyrical about how you must be extra careful:
System calls must carefully verify all their parameters to ensure that they are valid and
legal.The system call runs in kernel-space, and if the user can pass invalid input into the
kernel without restraint, the system’s security and stability can suffer.
and then provides a means for the user to totally annihilate the kernel :-)
The text from the copy I have states:
Let’s consider an example system call that uses both copy_from_user() and copy_to_user().This syscall, silly_copy(), is utterly worthless; it copies data from its first parameter into its second.This is suboptimal in that it involves an intermediate and extraneous copy into kernel-space for no gain. But it helps illustrate the point.
/*
* silly_copy - pointless syscall that copies the len bytes from
* ‘src’ to ‘dst’ using the kernel as an intermediary in the copy.
* Intended as an example of copying to and from the kernel.
*/
SYSCALL_DEFINE3(silly_copy,
unsigned long *, src,
unsigned long *, dst,
unsigned long len)
{
unsigned long buf;
/* copy src, which is in the user’s address space, into buf */
if (copy_from_user(&buf, src, len))
return -EFAULT;
/* copy buf into dst, which is in the user’s address space */
if (copy_to_user(dst, &buf, len))
return -EFAULT;
/* return amount of data copied */
return len;
}
Other than the catastrophic failure of not checking parameters, I'm pretty certain the last parameter of the SYSCALL_DEFINE3 is missing a comma (though that would just be a typo).
A far better example, without having to allocate arbitrary memory, would be along the lines of:
SYSCALL_DEFINE3(silly_copy,
unsigned long *, src,
unsigned long *, dst,
unsigned long, len)
{
unsigned long buf[64]; /* Buffer for chunks */
unsigned long lenleft = len; /* Remaining size */
unsigned long chunklen = sizeof(buf); /* Initial chunk length */
/* Loop handling chunk sizes */
while (lenleft > 0) {
/* Change chunk length on last chunk */
if (lenleft < chunklen) chunklen = lenleft;
/* copy src(user) to buf(kernel) then dst(user) */
if (copy_from_user(buf, src, chunklen)) return -EFAULT;
if (copy_to_user(dst, buf, chunklen)) return -EFAULT;
/* Adjust pointers and remaining size */
src += chunklen; dst += chunklen; lenleft -= chunklen;
}
/* return amount of data copied */
return len;
}
Anyone trying to implement that system call would be well advised to steer away from that particular sample in the book, although I suppose, at a bare minimum, it will give you some good kernel debugging experience :-)
int init_module(void)
{
mempool_t *mempool;
struct kmem_cache *kmem_cache;
void *p0 , *p1;
kmem_cache = kmem_cache_create("Ashrama" ,100 , 0 ,SLAB_PANIC ,NULL);
mempool = mempool_create(4 , mempool_alloc_slab , mempool_free_slab , kmem_cache);
p0 = mempool_alloc(mempool, SLAB_PANIC);
p1 = mempool_alloc(mempool , SLAB_PANIC);
strcpy(p0 , "Ranjan.B.M");
strcpy(p1 , "Mithun.V");
mempool_free( p0 , mempool);
printk(KERN_ALERT"%s",p0);
printk(KERN_ALERT"%s",p1);
}
Related
I'm trying to understand this code for reading virtual memory mappings but I'm having trouble understanding the different data types as I can't find any good documentation.
What is the difference between vm_offset_t, void *, and mach_vm_size_t? On my machine they all seem to be 8 bytes (64-bit) and used to navigate virtual memory. What are the differences between their purposes? What is the point of having these different types?
EDIT:
For instance, in the linked code:
unsigned char *
readProcessMemory (int pid, mach_vm_address_t addr, mach_msg_type_number_t *size)
{
// Helper function to read process memory (a la Win32 API of same name)
// To make it easier for inclusion elsewhere, it takes a pid, and
// does the task_for_pid by itself. Given that iOS invalidates task ports
// after use, it's actually a good idea, since we'd need to reget anyway
task_t t;
task_for_pid(mach_task_self(),pid, &t);
mach_msg_type_number_t dataCnt = size;
vm_offset_t readMem;
// Use vm_read, rather than mach_vm_read, since the latter is different
// in iOS.
kern_return_t kr = vm_read(t, // vm_map_t target_task,
addr, // mach_vm_address_t address,
*size, // mach_vm_size_t size
&readMem, //vm_offset_t *data,
size); // mach_msg_type_number_t *dataCnt
if (kr) {
// DANG..
fprintf (stderr, "Unable to read target task's memory #%p - kr 0x%x\n" , addr, kr);
return NULL;
}
return ( (unsigned char *) readMem);
}
According to this documentation of the vm_read function, the data_out parameter is an "Out-pointer to dynamic array of bytes returned by the read."
But in the code above they pass in &readMem (which is type vm_offset_t *) for data_out. I'm confused how `readMem is being used here - is it a pointer to the dynamic array of bytes returned by the read? Or does it actually contain the dynamic array of bytes? Is vm_offset_t a pointer or an address? What is its purpose?
Similarly
vm_offset_t, void*, and mach_vm_size_t are all internally synonymous with unsigned long, but they are used to make the code more readable and expressive.
vm_read returns an address in readMem, meaning that readMem will need to be cast to a pointer and dereferenced to access its value.
Also, the memory region pointed to by readMem is allocated by the kernel, so it needs to be deallocated with vm_deallocate. To avoid this, consider using vm_read_overwrite which will populate the buffer it is supplied.
My current concat function:
char* concat(char* a, int a_size,
char* b, int b_size) {
char* c = malloc(a_size + b_size);
memcpy(c, a, a_size);
memcpy(c + a_size, b, b_size);
free(a);
free(b);
return c;
}
But this used extra memory. Is it possible to append two byte arrays using realloc without making extra memory space?
Like:
void append(char* a, int a_size, char* b, int b_size)
...
char* a = malloc(2);
char* b = malloc(2);
void append(a, 2, b, 2);
//The size of a will be 4.
While Jean-François Fabre answered the stated question, I'd like to point out that you can manage such byte arrays better by using a structure:
typedef struct {
size_t max; /* Number of chars allocated for */
size_t len; /* Number of chars in use */
unsigned char *data;
} bytearray;
#define BYTEARRAY_INIT { 0, 0, NULL }
void bytearray_init(bytearray *barray)
{
barray->max = 0;
barray->len = 0;
barray->data = NULL;
}
void bytearray_free(bytearray *barray)
{
free(barray->data);
barray->max = 0;
barray->len = 0;
barray->data = NULL;
}
To declare an empty byte array, you can use either bytearray myba = BYTEARRAY_INIT; or bytearray myba; bytearray_init(&myba);. The two are equivalent.
When you no longer need the array, call bytearray_free(&myba);. Note that free(NULL) is safe and does nothing, so it is perfectly safe to free a bytearray that you have initialized, but not used.
To append to a bytearray:
int bytearray_append(bytearray *barray, const void *from, const size_t size)
{
if (barray->len + size > barray->max) {
const size_t len = barray->len + size;
size_t max;
void *data;
/* Example policy: */
if (len < 8)
max = 8; /* At least 8 chars, */
else
if (len < 4194304)
max = (3*len) / 2; /* grow by 50% up to 4,194,304 bytes, */
else
max = (len | 2097151) + 2097153 - 24; /* then pad to next multiple of 2,097,152 sans 24 bytes. */
data = realloc(barray->data, max);
if (!data) {
/* Not enough memory available. Old data is still valid. */
return -1;
}
barray->max = max;
barray->data = data;
}
/* Copy appended data; we know there is room now. */
memmove(barray->data + barray->len, from, size);
barray->len += size;
return 0;
}
Since this function can at least theoretically fail to reallocate memory, it will return 0 if successful, and nonzero if it cannot reallocate enough memory.
There is no need for a malloc() call, because realloc(NULL, size) is exactly equivalent to malloc(size).
The "growth policy" is a very debatable issue. You can just make max = barray->len + size, and be done with it. However, dynamic memory management functions are relatively slow, so in practice, we don't want to call realloc() for every small little addition.
The above policy tries to do something better, but not too aggressive: it always allocates at least 8 characters, even if less is needed. Up to 4,194,304 characters, it allocates 50% extra. Above that, it rounds the allocation size to the next multiple of 2,097,152 and substracts 24. The reasoning behid this is complex, but it is more for illustration and understanding than anything else; it is definitely NOT "this is best, and this is what you should do too". This policy ensures that each byte array allocates at most 4,194,304 = 222 unused characters. However, 2,097,152 = 221 is the size of a huge page on AMD64 (x86-64), and is a power-of-two multiple of a native page size on basically all architectures. It is also large enough to switch from so-called sbrk() allocation to memory mapping on basically all architectures that do that. It means that such huge allocations use a separate part of the heap for each, and the unused part is usually just virtual memory, not necessarily backed by any RAM, until accessed. As a result, this policy tends to work quite well for both very short byte arrays, and very long byte arrays, on most architectures.
Of course, if you know (or measure!) the typical size of the byte arrays in typical workloads, you can optimize the growth policy for that, and get even better results.
Finally, it uses memmove() instead of memcpy(), just in case someone wishes to repeat a part of the same byte array: memcpy() only works if the source and target areas do not overlap; memmove() works even in that case.
When using more advanced data structures, like hash tables, a variant of the above structure is often useful. (That is, this is much better in cases where you have lots of empty byte arrays.)
Instead of having a pointer to the data, the data is part of the structure itself, as a C99 flexible array member:
typedef struct {
size_t max;
size_t len;
unsigned char data[];
} bytearray;
You cannot declare a byte array itself (i.e. bytearray myba; will not work); you always declare a pointer to a such byte arrays: bytearray *myba = NULL;. The pointer being NULL is just treated the same as an empty byte array.
In particular, to see how many data items such an array has, you use an accessor function (also defined in the same header file as the data structure), rather than myba.len:
static inline size_t bytearray_len(bytearray *const barray)
{
return (barray) ? barray->len : 0;
}
static inline size_t bytearray_max(bytearray *const barray)
{
return (barray) ? barray->max : 0;
}
The (expression) ? (if-true) : (if-false) is a ternary operator. In this case, the first function is exactly equivalent to
static inline size_t bytearray_len(bytearray *const barray)
{
if (barray)
return barray->len;
else
return 0;
}
If you wonder about the bytearray *const barray, remember that pointer declarations are read from right to left, with * as "a pointer to". So, it just means that barray is constant, a pointer to a byte array. That is, we may change the data it points to, but we won't change the pointer itself. Compilers can usually detect such stuff themselves, but it may help; the main point is however to remind us human programmers that the pointer itself is not to be changed. (Such changes would only be visible within the function itself.)
Since such arrays often need to be resized, the resizing is often put into a separate helper function:
bytearray *bytearray_resize(bytearray *const barray, const size_t len)
{
bytearray *temp;
if (!len) {
free(barray);
errno = 0;
return NULL;
}
if (!barray) {
temp = malloc(sizeof (bytearray) + len * sizeof barray->data[0]);
if (!temp) {
errno = ENOMEM;
return NULL;
}
temp->max = len;
temp->len = 0;
return temp;
}
if (barray->len > len)
barray->len = len;
if (barray->max == len)
return barray;
temp = realloc(barray, sizeof (bytearray) + len * sizeof barray->data[0]);
if (!temp) {
free(barray);
errno = ENOMEM;
return NULL;
}
temp->max = len;
return temp;
}
What does that errno = 0 do in there? The idea is that because resizing/reallocating a byte array may change the pointer, we return the new one. If the allocation fails, we return NULL with errno == ENOMEM, just like malloc()/realloc() do. However, since the desired new length was zero, this saves memory by freeing the old byte array if any, and returns NULL. But since that is not an error, we set errno to zero, so that it is easier for callers to check if an error occurred or not. (If the function returns NULL, check errno. If errno is nonzero, an error occurred; you can use strerror(errno) to get a descriptive error message.)
You probably also noted the sizeof barray->data[0], used even when barray is NULL. This is okay, because sizeof is not a function, but an operator: it does not access the right side at all, it only evaluates to the size of the thing the right side refers to. (You only need to use parentheses when the right size is a type.) This form is nice, because it lets a programmer change the type of the data member, without changing any other code.
To append data to such a byte array, we probably want to be able to specify whether we anticipate further appends to the same array, or whether this is probably the final append, so that only the exact needed amount of memory is needed. For simplicity, I'll only implement the exact size version here. Note that this function returns a pointer to the (modified) byte array:
bytearray *bytearray_append(bytearray *barray,
const void *from, const size_t size,
int exact)
{
size_t len = bytearray_len(barray) + size;
if (exact) {
barray = bytearray_resize(barray, len);
if (!barray)
return NULL; /* errno already set by bytearray_resize(). */
} else
if (bytearray_max(barray) < len) {
if (!exact) {
/* Apply growth policy */
if (len < 8)
len = 8;
else
if (len < 4194304)
len = (3 * len) / 2;
else
len = (len | 2097151) + 2097153 - 24;
}
barray = bytearray_resize(barray, len);
if (!barray)
return NULL; /* errno already set by the bytearray_resize() call */
}
if (size) {
memmove(barray->data + barray->len, from, size);
barray->len += size;
}
return barray;
}
This time, we declared bytearray *barray, because we change where barray points to in the function. If the fourth parameter, final, is nonzero, then the resulting byte array is exactly the size needed; otherwise the growth policy is applied.
yes, since realloc will preserve the start of your buffer if the new size is bigger:
char* concat(char* a, size_t a_size,
char* b, size_t b_size) {
char* c = realloc(a, a_size + b_size);
memcpy(c + a_size, b, b_size); // dest is after "a" data, source is b with b_size
free(b);
return c;
}
c may be different from a (if the original memory block cannot be resized in-place contiguously to the new size by the system) but if that's the case, the location pointed by a will be freed (you must not free it), and the original data will be "moved".
My advice is to warn the users of your function that the input buffers must be allocated using malloc, else it will crash badly.
I have to write for my assignement a program that will consist of agents and a central server deamon. It will be a distributed shell - every command issued from a server will be also performed on every agent(the output will be sent back from every agent to central server).
I will have to deal with output commands (like ls -la /home/user/dir1) - on each agent the output may vary in size). The output of "find /" will also be BIG but I have to take somehow into account that something like that can happen. What is desired way of handling varying size outputs in C and operating on them? (saving to variable, sending it over a socket).
The way to deal with data of arbitrary size is to use dynamic allocation, i.e. the functions malloc(), realloc() and free(). You allocate and possibly grow the memory needed to store the command output.
Reading command output (assuming a Unix-like OS) is best done with popen().
Read the manuals of each of the mentioned functions for details.
Dynamic Memory Allocation
To hold your "variable length" strings, you should use dynamic memory allocation: the malloc family of functions.
#include <stdlib.h>
void *malloc(size_t size);
void free(void *ptr);
void *calloc(size_t nmemb, size_t size);
void *realloc(void *ptr, size_t size);
So, suppose you have your data stored in a variable char *ag_str. I suggest you malloc and then realloc the size of the buffer in blocks. Calling malloc and then realloc a thousand times to readjust the block size after each character is very costly.
So, you might do something like this:
#define BLOCK_SIZE 4096
struct mem_block {
size_t current_block_size;
size_t current_str_size;
char *ag_str;
};
struct mem_block *new_chunk(void)
{
struct mem_block *p = malloc(sizeof *p);
p->ag_str = malloc(BLOCK_SIZE);
p->current_block_size = BLOCK_SIZE;
p->current_str_size = 0;
return p;
}
void realloc_chunk(struct mem_block *chunk)
{
size_t ns = chunk->current_block_size + BLOCK_SIZE;
chunk->ag_str = realloc(chunk->ag_str, ns);
chunk->current_block_size = ns;
}
void cat_ag_str(struct mem_block *chunk, char *ag_str, size_t ag_len)
{
if (chunk->current_str_size + ag_len > chunk->current_block_size)
realloc_chunk(chunk);
strncat (chunk->ag_str, ag_str, ag_len);
chunk->current_str_size += ag_len;
}
void receive_from_agent(...)
{
struct mem_block *chunk = new_chunk();
ssize_t c; // Linux read/recv return
size_t count;
char buff[BLOCK_SIZE];
while((c = read(your_fd, buff, BLOCK_SIZE)) // or probably recv()
if (c < 0) ...
count = (size_t)c;
cat_ag_str(chunk, buff, count);
(...)
}
Note that this code was not tested and is just an idea for you. (Error checking was omitted)
struct mem_block: This will keep information about your current memory block.
new_chunk: function to create a new chunk handler for you.
realloc_chunk: anytime the amount of characters that must be written exceeds the amount of characters available in the chunk, we get one more block.
cat_ag_str: this will append what you just read to the memory block you have, effectively transforming chunks of data into one coherent big buffer.
receive_from_agent: this is the entry point of your receiving loop. You may use read or recv, I don't know which you use, but both return the amount of bytes read, which you'll use to pass to cat_ag_str.
It's important to note that you're reading in the same sized blocks as you realloc. (You can read in smaller chunks too, but never bigger).
You can do roughly the same for sending, but you don't need all that workaround for memory. You can just use a fixed sized buffer and copy data from your big string to it in fixed sizes, then you send the fixed-sized buffer.
I have a function in C that dynamically allocates a buffer, which is passed to another function to store its return value. Something like the following dummy example:
void other_function(float in, float *out, int out_len) {
/* Fills 'out' with 'out_len' values calculated from 'in' */
}
void function(float *data, int data_len, float *out) {
float *buf;
int buf_len = 2 * data_len, i;
buf = malloc(sizeof(float) * buf_len);
for (i = 0; i < data_len; i++, data++, out++) {
other_function(*data, buf, buf_len);
/* Do some other stuff with the contents of buf and write to *out */
}
free buf;
}
function is called by an iterator over a multi-dimensional array (it's a NumPy gufunc kernel, to be precise), so it gets called millions of times with the same value for data_len. It seems wasteful to be creating and destroying the buffer over and over again. I would normally move allocation of the buffer to the function that calls function, and pass a poiinter to it, but I don't directly control that, so not possible. Instead, I am considering doing the following:
void function(float *data, int data_len, float *out) {
static float *buf = NULL;
static int buf_len = 0;
int i;
if (buf_len != 2 * data_len) {
buf_len = 2 * data_len;
buf = realloc(buf, sizeof(float) * buf_len); /* same as malloc if buf == NULL */
}
for (i = 0; i < data_len; i++, data++, out++) {
other_function(*data, buf, buf_len);
/* Do some other stuff with the contents of buf and write to *out */
}
}
That means that I never directly free the memory I allocate: it gets reused in subsequent calls, and then lingers there until my program exits. It doesn't seem like the right thing to do, but not too bad either, as the amount of memory allocated is always going to be small. Am I over worrying? Is there a better approach to this?
This approach is legitimate (but see below), although tools like valgrind will incorrectly flag it as a "leak". (It's not a leak, as a leak is an unbounded increase in memory usage.) You might want to benchmark exactly how much time is lost on malloc and free compared to other things the function is doing.
If you can use C99 or gcc, and if your buffer is not overly large, you should also consider variable-length arrays, which are as fast (or faster than) a static buffer, and create no fragmentation. If you're on another compiler, you can look into the non-standard (but widely supported) alloca extension.
You do need to be aware that using a static buffer makes your function:
Thread-unsafe - if it is called from multiple threads simultaneously, it will destroy the data of the other instance. If the Python is called from numpy, this is probably not a problem, as threads will be effectively serialized by the GIL.
Non-reentrant - if other_function calls some Python code which ends up calling function - for whatever reason - before function finishes, your function will again destroy its own data.
If you don't need true parallel execution and reentrancy, this use of static variables is fine, and a lot of C code uses them that way.
This is a fine approach, and something like this is likely used internally by many libraries. The memory will be freed automatically when the program exits.
You might want to round buf_len up to a multiple of some block size, so you don't realloc() every time data_len changes a small bit. But if data_len is almost always the same size, this isn't necessary.
I am trying to add data received onto a buffer which needs to be configurable at runtime (I read a size from file or command line).
So basically I determine my buffersize and allocate an area of memory using calloc (I also put a catchall to set a buffersize if it is not in the config file or command line - Let's assume we use that for now).
I am only putting applicable lines of code.
int buffersize=10000;
void *BuffPtr = (void *)calloc(1,buffersize * sizeof(char));
I then have a recv from UDP (I have tried receiving into char array and dynamically allocated array - both work fine)
// Setup socket......
void *PktBuff = (void *)calloc(1,1000 * sizeof(char));
// Loop and receive many packets......
rcvd_bytes=recv(recv_socket, PktBuff, 1000, 0);
I can, at this point, write the contents of PktBuff and it works fine. But I want to concatenate a number of received packets in my dynamically allocated array (BuffPtr defined above).
I have tried strcat, but I just get garbage out if I try to write the first packet received, without getting another packet.
strcat(BuffPtr, PktBuff);
What I am doing wrong??
Thanks in advance.
Your data doesn't seem to be 0-terminated strings, you may want to use memmove instead.
A few points and observations:
Don't cast the return value of malloc() in C.
The expression sizeof (char) is a wordy way of writing 1, multiplying by it is seldom informative.
Make sure all your data is 0-terminated (strings), otherwise you can't use string functions since that's what they require.
You should probably just use an extra size_t counter to keep track of the number of bytes in BuffPtr, and use that and memcpy() to append.
Okay a number of issues in your code strcat is NOT the right way to append binary data. Here is a semi-robust implementation. I have NOT checked it for syntax errors, just typed it out please study it as an example, and adjust it to your code.
/* total number of bytes you are willing to receive in single a single receive */
#define MAX_RECV_BUFFER 1000
/* total number of bytes you are willing to store in memory */
#define MAXBYTES MAX_RECV_BUFFER*1000
int append_bytes() {
char rcvbuf[MAX_RECV_BUFFER]; /* buffer where things are received */
void *buf = NULL; /* buffer where bytes are collected */
size_t rcvlen; /* length of data received */
size_t buflen = 0; /* total bytes */
while(1) {
void *p;
rcvlen = recv(recv_socket, rcvbuf, MAX_RECV_BUFFER, 0);
if ( rcvlen < 0 ) { manage_error(); }
if ( rcvlen == 0 )
break;
if ( buflen + rcvlen > MAXBYTES)
break;
p = realloc(buf, buflen+recvlen);
if ( !p ) { manage_memory_error(); }
buf = p; /* we have enough space */
memcpy(buf+buflen, rcvbuf, recvlen); /* append new content to end of buffer */
buflen+=recvlen; /* add length to buflen */
}
}