So I'm writing a linux kernel module that involves writing to a proc file. Unfortunately something is going wrong with the newline character. If I open it with vim, it shows as "num^#num^#num^#". If I cat it, it says "numnumnum". It should go to a new line at the end of each "num".
My code for writing each entry to the proc file admittedly seems kind of hacky.
bufSize = snprintf(str,0,"%lu\n",var);
str = (char*)kmalloc(bufSize*sizeof(char),GFP_KERNEL);
snprintf(str,bufSize,"%lu\n",var);
memcpy(msg+msglen,str,bufSize);
msglen+=(bufSize);
kfree(str);
I don't know how long the string will be, so the first snprintf gets the length needed for the buffer. The buffer is initialized, then snprintf is called again. The string is then copied to msg, which contains the data for the proc file. The pointer is incremented by the length of the existing message.
int procfile_read(char *buffer, char **buffer_location, off_t offset, int
buffer_length, int *eof, void *data) {
int ret;
printk(KERN_INFO "procfile_read (/proc/%s) called\n", PROCFS_NAME);
if (offset > 0) {
/* we have finished to read, return 0 */
ret = 0;
} else {
/* fill the buffer, return the buffer size */
memcpy(buffer, msg, msglen);
ret = msglen;
}
return ret;
This is pretty much copied and pasted from a tutorial.
Thanks!
Buffer size is too small
bufSize = snprintf(str,0,"%lu\n",var);
// + 1
str = (char*)kmalloc((bufSize + 1)*sizeof(char),GFP_KERNEL);
// + 1
snprintf(str,bufSize + 1,"%lu\n",var);
// + 1
memcpy(msg+msglen,str,bufSize + 1);
// no + 1 here
// Note that msglen is the string length. Add 1 for the size needed.
msglen+=(bufSize);
kfree(str);
Related
Documentation for pb_ostream_from_buffer says
After writing, you can check stream.bytes_written to find out how much
valid data there is in the buffer. This should be passed as the
message length on decoding side.
So ideally, when I send the serialized data I need to also send the bytes_written as a parameter separate from the buffer.
The problem is that my interface only allows me to send one variable: the buffer.
QUESTION
How do I specify always serialize the struct with no optimizations so that bufsize in
pb_istream_from_buffer(const pb_byte_t *buf, size_t bufsize)
can be a constant (i.e. the macro that specifies the maximum size) instead of needing to pass stream.bytes_written?
According to the Protocol Buffers encoding specification there are variable size types (like int32, int64, string, etc) and fixed size types (like fixed32, fixed64, double, etc). Now, this variable size encoding is more than just an optimization, it's a part of the design and specification. So disabling this "optimization" by the means of Protocol Buffers is only possible if your data consists exclusively of fixed length types and has no repeated fields as long as the number of repetitions is not fixed. I presume that this is not the case, since you're asking this question. So the short answer is no, it's not possible by means of the library because it would violate the encoding specification.
But in my opinion the desired effect could be easily achieved by encoding the size into the buffer with little CPU and RAM overhead. I presume you know the maximum size of the message generated by nanopb, we denote it by MAX_MSG_SIZE. We call this message the payload message. Suppose that this MAX_MSG_SIZE can be represented by some integer type, which we denote by wrapped_size_t (e.g. uint16_t).
The idea is simple:
allocate the buffer slightly larger than MAX_MSG_SIZE;
write the payload message generated by nanopb at some offset into the allocated buffer;
use this offset to encode the size of the payload message at the beginning of the buffer;
transmit the whole buffer having the fixed size equal to MAX_MSG_SIZE + sizeof(wrapped_size_t) to the receiver;
upon reception decode the size of the payload message and pass both the decoded size and the payload message to pb_istream_from_buffer.
I attach the code to illustrate the idea. I used an example from nanopb repository:
#include <stdio.h>
#include <inttypes.h>
#include <string.h>
#include <pb_encode.h>
#include <pb_decode.h>
#include "simple.pb.h"
//#define COMMON_ENDIANNES
#ifdef COMMON_ENDIANNES
#define encode_size encode_size_ce
#define decode_size decode_size_ce
#else
#define encode_size encode_size_le
#define decode_size decode_size_le
#endif
typedef uint16_t wrapped_size_t;
/* Maximum size of the message returned by bytes_written */
const size_t MAX_MSG_SIZE = 11;
/* Size of the field storing the actual size of the message
* (as returned by bytes_written) */
const size_t SIZE_FIELD = sizeof(wrapped_size_t);
/* Fixed wrapped message size */
const size_t FIXED_MSG_SIZE = MAX_MSG_SIZE + sizeof(wrapped_size_t);
void print_usage(char *prog);
/* Get the address of the payload buffer from the transmitted buffer */
uint8_t* payload_buffer(uint8_t *buffer);
/* Encode the payload size into the transmitted buffer (common endiannes) */
void encode_size_ce(uint8_t *buffer, size_t size);
/* Decode the payload size into the transmitted buffer (common endiannes) */
wrapped_size_t decode_size_ce(uint8_t *buffer);
/* Encode the payload size into the transmitted buffer (little endian) */
void encode_size_le(uint8_t *buffer, size_t size);
/* Decode the payload size into the transmitted buffer (little endian) */
size_t decode_size_le(uint8_t *buffer);
int main(int argc, char* argv[])
{
/* This is the buffer where we will store our message. */
uint8_t buffer[MAX_MSG_SIZE + sizeof(wrapped_size_t)];
bool status;
if(argc > 2 || (argc == 2 && (!strcmp(argv[1], "-h") || !strcmp(argv[1], "--help"))))
{
print_usage(argv[0]);
return 1;
}
/* Encode our message */
{
/* Allocate space on the stack to store the message data.
*
* Nanopb generates simple struct definitions for all the messages.
* - check out the contents of simple.pb.h!
* It is a good idea to always initialize your structures
* so that you do not have garbage data from RAM in there.
*/
SimpleMessage message = SimpleMessage_init_zero;
/* Create a stream that will write to our buffer. */
pb_ostream_t stream = pb_ostream_from_buffer(payload_buffer(buffer),
MAX_MSG_SIZE);
if(argc > 1)
sscanf(argv[1], "%" SCNd32, &message.lucky_number);
else
{
printf("Input lucky number: ");
scanf("%" SCNd32, &message.lucky_number);
}
/* Encode the payload message */
status = pb_encode(&stream, SimpleMessage_fields, &message);
/* Wrap the payload, i.e. add the size to the buffer */
encode_size(buffer, stream.bytes_written);
/* Then just check for any errors.. */
if (!status)
{
printf("Encoding failed: %s\n", PB_GET_ERROR(&stream));
return 1;
}
}
/* Now we could transmit the message over network, store it in a file, etc.
* Note, the transmitted message has a fixed length equal to FIXED_MSG_SIZE
* and is stored in buffer
*/
/* But for the sake of simplicity we will just decode it immediately. */
{
/* Allocate space for the decoded message. */
SimpleMessage message = SimpleMessage_init_zero;
/* Create a stream that reads from the buffer. */
pb_istream_t stream = pb_istream_from_buffer(payload_buffer(buffer),
decode_size(buffer));
/* Now we are ready to decode the message. */
status = pb_decode(&stream, SimpleMessage_fields, &message);
/* Check for errors... */
if (!status)
{
printf("Decoding failed: %s\n", PB_GET_ERROR(&stream));
return 1;
}
/* Print the data contained in the message. */
printf("Your lucky number was %d; payload length was %d.\n",
(int)message.lucky_number, (int)decode_size(buffer));
}
return 0;
}
void print_usage(char *prog)
{
printf("usage: %s [<lucky_number>]\n", prog);
}
uint8_t* payload_buffer(uint8_t *buffer)
{
return buffer + SIZE_FIELD;
}
void encode_size_ce(uint8_t *buffer, size_t size)
{
*(wrapped_size_t*)buffer = size;
}
wrapped_size_t decode_size_ce(uint8_t *buffer)
{
return *(wrapped_size_t*)buffer;
}
void encode_size_le(uint8_t *buffer, size_t size)
{
int i;
for(i = 0; i < sizeof(wrapped_size_t); ++i)
{
buffer[i] = size;
size >>= 8;
}
}
size_t decode_size_le(uint8_t *buffer)
{
int i;
size_t ret = 0;
for(i = sizeof(wrapped_size_t) - 1; i >= 0; --i)
ret = buffer[i] + (ret << 8);
return ret;
}
UPD Ok, if, for some reason, you still wish to stick to the original GPB encoding there's another option available: fill the unused part of the buffer (i.e. the part after the last byte written by nanopb) with some valid data which will be ignored. For instance, you can reserve a field number which doesn't mark any field in your *.proto file but is used to mark the data which will be discarded by the GPB decoder. Let's denote this reserved field number as RESERVED_FIELD_NUMBER. This is used for backward compatibility but you can use it for your purpose as well. Let's call this filling-in the buffer with the dummy data sealing (perhaps there's a better term). This method also requires that you have at least 2 free bytes available to you after pb_encode.
So the idea of sealing is even simpler:
calculate how many buffer bytes is left unfilled after pb_encode;
mark the rest of the buffer as array of bytes with RESERVED_FIELD_NUMBER.
I attach the updated code, the main function is bool seal_buffer(uint8_t *buffer, size_t size), call it after pb_encode to seal the buffer and you're done. Currently, it has a limitation of sealing no more than 2 ** 28 + 4 bytes, but it could be easily updated to overcome this limitation.
#include <stdio.h>
#include <assert.h>
#include <inttypes.h>
#include <pb_encode.h>
#include <pb_decode.h>
#include "simple.pb.h"
/* Reserved field_number shouldn't be used for field numbering. We use it
* to mark the data which will be ignored upon reception by GPB parser.
* This number should be 1 to 15 to fit into a single byte. */
const uint8_t RESERVED_FIELD_NUMBER = 15;
/* Maximum size of the message returned by bytes_written (payload size) */
const size_t MAX_MSG_SIZE = 200;
/* Size of the transmitted message (reserve 2 bytes for minimal sealing) */
const size_t FIXED_MSG_SIZE = MAX_MSG_SIZE + 2;
void print_usage(char *prog);
/* Sealing the buffer means filling it in with data which is valid
* in the sense that a GPB parser accepts it as valid but ignores it */
bool seal_buffer(uint8_t *buffer, size_t size);
int main(int argc, char* argv[])
{
/* This is the buffer where we will store our message. */
uint8_t buffer[FIXED_MSG_SIZE];
bool status;
if(argc > 2 || (argc == 2 && (!strcmp(argv[1], "-h") || !strcmp(argv[1], "--help"))))
{
print_usage(argv[0]);
return 1;
}
/* Encode our message */
{
/* Allocate space on the stack to store the message data.
*
* Nanopb generates simple struct definitions for all the messages.
* - check out the contents of simple.pb.h!
* It is a good idea to always initialize your structures
* so that you do not have garbage data from RAM in there.
*/
SimpleMessage message = SimpleMessage_init_zero;
/* Create a stream that will write to our buffer. */
pb_ostream_t stream = pb_ostream_from_buffer(buffer, sizeof(buffer));
if(argc > 1)
sscanf(argv[1], "%" SCNd32, &message.lucky_number);
else
{
printf("Input lucky number: ");
scanf("%" SCNd32, &message.lucky_number);
}
/* Now we are ready to encode the message! */
status = pb_encode(&stream, SimpleMessage_fields, &message);
/* Then just check for any errors.. */
if (!status)
{
fprintf(stderr, "Encoding failed: %s\n", PB_GET_ERROR(&stream));
return 1;
}
/* Now the main part - making the buffer fixed-size */
assert(stream.bytes_written + 2 <= FIXED_MSG_SIZE);
if(!seal_buffer(buffer + stream.bytes_written,
FIXED_MSG_SIZE - stream.bytes_written))
{
fprintf(stderr, "Failed sealing the buffer "
"(filling in with valid but ignored data)\n");
return 1;
}
}
/* Now we could transmit the message over network, store it in a file or
* wrap it to a pigeon's leg.
*/
/* But because we are lazy, we will just decode it immediately. */
{
/* Allocate space for the decoded message. */
SimpleMessage message = SimpleMessage_init_zero;
/* Create a stream that reads from the buffer. */
pb_istream_t stream = pb_istream_from_buffer(buffer, FIXED_MSG_SIZE);
/* Now we are ready to decode the message. */
status = pb_decode(&stream, SimpleMessage_fields, &message);
/* Check for errors... */
if (!status)
{
fprintf(stderr, "Decoding failed: %s\n", PB_GET_ERROR(&stream));
return 1;
}
/* Print the data contained in the message. */
printf("Your lucky number was %d.\n", (int)message.lucky_number);
}
return 0;
}
void print_usage(char *prog)
{
printf("usage: %s [<lucky_number>]\n", prog);
}
bool seal_buffer(uint8_t *buffer, size_t size)
{
size_t i;
if(size == 1)
{
fprintf( stderr, "Cannot seal the buffer, at least 2 bytes are needed\n");
return false;
}
assert(size - 5 < 1<<28);
if(size - 5 >= 1<<28)
{
fprintf( stderr, "Representing the size exceeding 2 ** 28 + 4, "
"although it's not difficult, is not yet implemented\n");
return false;
}
buffer[0] = (15 << 3) + 2;
/* encode the size */
if(size - 2 < 1<<7)
buffer[1] = size - 2;
else
{
/* Size is large enough to fit into 7 bits (1 byte).
* For simplicity we represent the remaining size by 4 bytes (28 bits).
* Note that 1 byte is used for encoding field_number and wire_type,
* plus 4 bytes for the size encoding, therefore the "remaining size"
* is equal to (size - 5)
*/
size -= 5;
for(i = 0; i < 4; ++i)
{
buffer[i + 1] = i < 3? (size & 0x7f) | 0x80: size & 0x7f;
size >>= 7;
}
}
return true;
}
I have written a read function to read a file into buffer in Kernel space.
int readfile(const char *filename, void *buf, int len, int offset)
{
struct file *filp;
mm_segment_t oldfs;
int bytes;
filp = NULL;
filp = filp_open(filename, O_RDONLY, 0);
if(!filp || IS_ERR(filp)) {
printk(" Error in reading file %s. Error = %d\n", filename, \
(int) PTR_ERR(filp));
return -1;
}
filp->f_pos = offset;
oldfs = get_fs();
set_fs(get_ds());
bytes = vfs_read(filp, buf, len, &filp->f_pos);
set_fs(oldfs);
filp_close(filp, NULL);
return bytes;
}
Now, this function works really well and I am able to read contents of filename into buf by calling this function from my system call
char *firstbuffer;
firstbuffer = kmalloc(sizeof(PAGE_SIZE), GFP_KERNEL);
bytesread = readfile(firstfile, firstbuffer, len, 0);
// Null terminate read string
firstbuffer[bytesread] = '\0';
printk("first buffer = %s\n",firstbuffer);
Then, I am calling this function again to read contents of secondfile into secondbuffer.
char *secondbuffer;
secondbuffer = kmalloc(sizeof(PAGE_SIZE), GFP_KERNEL);
bytesread2 = readfile(secondfile, secondbuffer, len, 0);
// Null terminate read string
secondbuffer[bytesread2] = '\0';
printk("second buffer %s", secondbuffer);
The problem is that after calling the read function on secondfile, the contents of my firstbuffer are getting overridden with contents of secondbuffer.
For example: if the contents of firstfile are
A
B
C
and contents of secondfile are
X
Y
Z
then after first read file call, the content of firstbuffer is:
A
B
C
and then after second read file call, the content of firstbuffer is:
A
X
Y
Z
Now, I am not sure what is going wrong here, but after second read function call, contents of firstbuffer is getting merged with contents of secondbuffer. How do I fix this?
Disclaimer:
I know we shouldn't do file I/O in Kernel space. This is purely to learn how read functions work in Kernel space.
kmalloc(sizeof(PAGE_SIZE), GFP_KERNEL)
This allocates sizeof(PAGE_SIZE) bytes. Now, PAGE_SIZE is an integer, so it's probably 4 bytes long, so you allocate 4 bytes.
If you wanted to allocate PAGE_SIZE bytes, use:
kmalloc(PAGE_SIZE, GFP_KERNEL)
I'm having some difficulties extracting data from a buffer using memcpy.
First, I memcpy some variables into a buffer:
int l1_connect(const char* hostname, int port) {
// Variables to be stored in the buffer
char *msg = "Hi, I'm a message"; // strlen(msg) == 17
uint16_t sender_id = htons(1); // sizeof(sender_id) == 2
uint16_t packet_size = htons(sizeof(packet_size)+sizeof(sender_id)+strlen(msg)); // sizeof(packet_size) == 2
// Checking values
printf("l1_connect():\nsender_id: %d, packet_size: %d\n\n", ntohs(sender_id), ntohs(packet_size));
// sender_id == 1, packet_size == 21
// The buffer
char buf[100];
// Copying everything
memcpy(&buf, &sender_id, sizeof(sender_id));
memcpy(&buf+sizeof(sender_id), &packet_size, sizeof(packet_size));
memcpy(&buf+sizeof(sender_id)+sizeof(packet_size), &msg, strlen(msg));
// Passing buf to another function
int bytes_sent = l1_send(1, buf, sizeof(buf));
}
I then try to extract that data (checking, before sending over UDP socket):
int l1_send( int device, const char* buf, int length ) {
// Variables in which to store extracted data
uint16_t id = 0;
uint16_t size = 0;
char msg[50];
memcpy(&id, &buf, sizeof(id));
memcpy(&size, &buf+sizeof(id), sizeof(size));
int remaining = ntohs(size) - (sizeof(id) + sizeof(size));
printf("l1_send():\nremaining: %d\n", remaining); // -37041
// memcpy-ing with correct(?) offset
memcpy(&msg, &buf+sizeof(id)+sizeof(size), 50);
msg[49] = '\0';
printf("id: %d\n", ntohs(id)); // 8372
printf("size: %d\n", ntohs(size)); // 37045
printf("msg: %s\n", msg); // ��$_�
return 0; // For now
}
As you can see, the values aren't quite what I'm expecting. Can anyone tell me what I'm doing wrong?
Your pointer math is incorrect. You're using &buf where you should just be using buf. If this doesn't explain what is wrong, nothing else I can say will:
#include <stdio.h>
int main(int argc, char **argv)
{
char buff[100];
printf("buff : %p\nbuff+10 : %p\n&buff+10 : %p\n", buff, buff+10, &buff+10);
return 0;
}
Output (varies by platform, obviously)
buff : 0xbf87a8bc
buff+10 : 0xbf87a8c6
&buff+10 : 0xbf87aca4
See it live. The math you're doing is incrementing by type, which for &buf is a pointer to array of 100 chars; not a simple char address. Therefore, &buff + 10 (in my sample) says "give me the 10th array of 100 chars from where I am now.". The subsequent write is invoking undefined behavior as a consequence.
Valgrind is your buddy here, btw. It would have caught this in a heartbeat.
Update
May as well fill in the entire gambit while I'm here. This is also wrong in l1_send:
memcpy(&id, &buf, sizeof(id));
// this------^
and the subsequent other areas you're using it in that function. You're taking the address of a parameter pointer, not the value within it. I'm confident you need buf there as well.
Try this:
memcpy(buf, &sender_id, sizeof(sender_id));
memcpy(buf + sizeof(sender_id), &packet_size, sizeof(packet_size));
memcpy(buf + sizeof(sender_id) + sizeof(packet_size), msg, strlen(msg));
To help you understand what is wrong with your code, you can read this.
Related: Pointer math vs. Array index
I have a structure where I stored some values as shown below:
struct cmd {
char *pname;
char *pdesc;
};
Following initialization I made as:
struct cmd[] = {{"show", "show items"},
{"exit", "exit the shell"},
{"setitem", "setting item"}
};
I'm using sprinf() to print by storing all the pname ans pdesc as below,
int length = 0;
char *resultcmd;
for (indx = 0; indx< cmdCount; indx++) {
length += sprintf(resultcmd+length, cmd[indx].pname, cmd[indx].pdesc);
}
Please help me how to allocate memory for resultcmd, It worked when i make resulecmd as array of some length, but if more pname and pdesc are added buffer overruns. Please help me.
If you want safely output data to buffer resultcmd you have to find out its length before and use it:
size_t length = 1; // 1 symbol needed to store \0',
// because strlen() returns length
// without NULL-termination symbol
// compute length:
for (intx = 0; indx < cmdCount; indx++) {
length += strlen(cmd[indx].pname) + strlen(cmd[indx].pdesc);
}
char *resultcmd = malloc(length);
int written = 0, ret = 0;
// print cmds to C string
for (indx = 0; indx < cmdCount; indx++) {
ret = snprintf (resultcmd + written, length - written,
"%s%s", cmd[indx].pname, cmd[indx].pdesc))
if (0 > ret) {
fprintf (stderr, "snprintf() error: %s\n", strerror(errno));
break;
} else {
written += ret;
}
}
/*
* some useful code here
*/
free(resultcmd);
You can use snprintf(char *dest, size_t maxlen, char *fmt, ...) to bound the size of your print. If the function fails, it returns the number of characters that would have been written, had there been enough space; so that +1 is what you need to realloc.
I get an
malloc: *** error for object 0x1001012f8: incorrect checksum for freed object
- object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
error in the following function:
char* substr(const char* source, const char* start, const char* end) {
char *path_start, *path_end, *path;
int path_len, needle_len = strlen(start);
path_start = strcasestr(source, start);
if (path_start != NULL) {
path_start += needle_len;
path_end = strcasestr(path_start, end);
path_len = path_end - path_start;
path = malloc(path_len + 1);
strncpy(path, path_start, path_len);
path[path_len] = '\0';
} else {
path = NULL;
}
return path;
}
How can I make this work? When I rewrite the function to allocate the memory using path[path_len + 1] it works just fine.
Now, the part I don't understand is, that I never even call free in any point of my application, as every allocated memory is needed for the program until it exists (which, AFAIK will invalidate every allocated memory anyway?!)
So, how can a freed object be corrupt if I never free one?
The function is called in this one:
char *read_response(int sock) {
int bytes_read;
char *buf = (char*)malloc(BUF_SIZE);
char *cur_position = buf;
while ((bytes_read = read(sock, cur_position, BUF_SIZE)) > 0) {
cur_position += bytes_read;
buf = realloc(buf, sizeof(buf) + BUF_SIZE);
}
int status = atoi(substr(buf, "HTTP/1.0 ", " "));
There is the realloc, am I using that wrong? I want to read the complete server response, so I have to reallocate after every iteration, don't I?
In read_response, you are probably overwriting the end of the buffer pointed to by buf.
The problem is that buf is a pointer, so sizeof(buf) will return the size of a pointer (probably 4 or 8 depending on your CPU). You are using sizeof as if buf were an array, which is not really the same thing as a pointer in C although they seem interchangeable in some contexts.
Instead of using sizeof, you need to be keeping track of the last size that you allocated for buf, and add BUF_SIZE to that each time you enlarge the buffer.
You should also consider that the read operation may be returning considerably fewer characters than BUF_SIZE on each call, so doing a realloc on buf in each iteration may be overkill. That probably won't cause any problems for you in terms of correctness, though; it will just use more memory than it needs to.
I would do something more like the code below.
#define MIN_BUF_SPACE_THRESHOLD (BUF_SIZE / 2)
char *read_response(int sock) {
int bytes_read;
char *buf = (char*)malloc(BUF_SIZE);
int cur_position = 0;
int space_left = BUF_SIZE;
if (buf == NULL) {
exit(1); /* or try to cope with out-of-memory situation */
}
while ((bytes_read = read(sock, buf + cur_position, space_left)) > 0) {
cur_position += bytes_read;
space_left -= bytes_read;
if (space_left < MIN_BUF_SPACE_THRESHOLD) {
buf = realloc(buf, cur_position + space_left + BUF_SIZE);
if (buf == NULL) {
exit(1); /* or try to cope with out-of-memory situation */
}
space_left += BUF_SIZE;
}
}
This version has the advantage of not trying to allocate more space if the read call comes back with only a few bytes of data.
This line
buf = realloc(buf, sizeof(buf) + BUF_SIZE);
is wrong. All reallocations are with the same size, BUF_SIZE + sizeof(char*). Then you are writing to unallocated memory when reading from the socket, overwriting memory previously freed by a realloc.
You have to keep track of the allocated size,
size_t current_buf_size = BUF_SIZE;
/* ... */
char *temp = realloc(buf, current_buf_size + BUF_SIZE);
if (temp == NULL) {
/* die or repair */
}
buf = temp;