The question may seems strange but I didn't misspelled it:
I want to decompress some data I downloaded without write them on the HDD. For that, I download it in a dynamically allocated buffer and I'd like to send it to the zlib wrapper I use (miniunzip). The problem is that this implementation is quite long (2-3K lines) and I'd like to avoid to have to rewrite it just for few lines.
I'd like to know if there was any way to read the buffer via a FILE* structure (miniunzip use it's own structure but I found a "fopen()" hiden under the loader). I know its length if it can help.
Thanks in advance and excuse me for my poor grammar.
I'm working on both Windows and UNIX systems (OSX/GNU Linux).
If you're talking about the minizip library that's included with zlib, you could use the unzOpen2 function which allows you to specify a structure containing the I/O functions to use. This should get you started:
struct zmem_data {
char *buf;
size_t length;
};
static voidpf zmemopen(voidpf opaque, const char *filename, int mode) {
if ((mode&ZLIB_FILEFUNC_MODE_READWRITEFILTER) != ZLIB_FILEFUNC_MODE_READ) return NULL;
uLong *pos = malloc(sizeof(uLong));
*pos = 0;
return pos;
}
static uLong zmemread(voidpf opaque, voidpf stream, void* buf, uLong size) {
struct zmem_data *data = (struct zmem_data*)opaque;
uLong *pos = (uLong*)stream;
uLong remaining = data->length - *pos;
uLong readlength = size < remaining ? size : remaining;
if (*pos > data->length) return 0;
memcpy(buf, data->buf+*pos, readlength);
*pos += readlength;
return readlength;
}
static uLong zmemwrite(voidpf opaque, voidpf stream, const void *buf, uLong size) {
/* no write support for now */
return 0;
}
static int zmemclose(voidpf opaque, voidpf stream) {
free(stream);
return 0;
}
static int zmemerror(voidpf opaque, voidpf stream) {
if (stream == NULL) return 1;
else return 0;
}
static long zmemtell(voidpf opaque, voidpf stream) {
return *(uLong*)stream;
}
static long zmemseek(voidpf opaque, voidpf stream, uLong offset, int origin) {
struct zmem_data *data = (struct zmem_data*)opaque;
uLong *pos = (uLong*)stream;
switch (origin) {
case ZLIB_FILEFUNC_SEEK_SET:
*pos = offset;
break;
case ZLIB_FILEFUNC_SEEK_CUR:
*pos = *pos + offset;
break;
case ZLIB_FILEFUNC_SEEK_END:
*pos = data->length + offset;
break;
default:
return -1;
}
return 0;
}
static void init_zmemfile(zlib_filefunc_def *inst, char *buf, size_t length) {
struct zmem_data *data = malloc(sizeof(struct zmem_data));
data->buf = buf;
data->length = length;
inst->opaque = data;
inst->zopen_file = zmemopen;
inst->zread_file = zmemread;
inst->zwrite_file = zmemwrite;
inst->ztell_file = zmemtell;
inst->zseek_file = zmemseek;
inst->zclose_file = zmemclose;
inst->zerror_file = zmemerror;
}
static void destroy_zmemfile(zlib_filefunc_def *inst) {
free(inst->opaque);
inst->opaque = NULL;
}
void example() {
zlib_filefunc_dec fileops;
init_zmemfile(&fileops, buffer, buffer_length);
unzFile zf = unzOpen2(NULL, &fileops);
/* ... process zip file ... */
unzClose(zf);
destroy_zmemfile(&fileops);
}
To summarize your question: You want to provide a FILE* interface for an in-memory buffer. No, you can't do that. The fread(), etc. calls actually end up making system calls that deal with an open file descriptor which you don't have.
You're over-complicating this. Decompression code almost always works from buffers that are in memory. If they have a file interface, certainly that is just a wrapper which handles reading the file into memory and then decompressing (probably in chunks to save memory). You should be able to find a decompression library with calls to decompress a buffer that you give it (just a pointer and a length).
Why you don't want to write the downloaded data to the hard drive is certainly up to you, but I hope this is for good, and not for evil.
The other option would be to open a memory-mapped file, to which you write during download, and read during decompression. There may be a way to specify that the file not be written to disk but of that I am unsure. Also, this would be very different between windows and linux.
These may help:
File Mapping (Windows)
mmap (POSIX)
Related
I have to implement a contructor function in c based upon this struct:
struct Node {
char name[MAX_NAME_LEN + 1];
NodeType type;
union {
struct {
Entry* entries; // list of directory entries
} dir;
struct {
void* contents; // any binary data of the given length
int length;
} file;
};
};
Now I want to build a constructor function to file, point to the contents and save the length in bytes but somehow I don't know how to do it...
Here is my Attempt:
i KNOW that i have to allocate space for this but how do i make this pointer stuff?
Node* new_file(char* name) {
Node* ptr_file = xmalloc(sizeof(Node));
ptr_file->name;
return NULL;
}
You need to typedef struct { ... } Node for your code to compile.
When using anonymous structs/unions you simply don't give a name for the anonymous member. ptr_file->dir.entries or ptr_file->file.contents.
Should you ditch the internal struct names and make those anonymous as well you would only need to type ptr_file->entries;.
Please note that anonymous structs/unions were added to the C language in the ISO 9899:2011 version of the language, so you need to use a fairly modern compiler to use them.
As a side note, the void* probably doesn't make any sense. What I think that you are trying to do is this:
#include <stdint.h>
typedef struct {
char name[MAX_NAME_LEN + 1];
NodeType type;
union {
struct {
Entry* entries; // list of directory entries
};
struct {
uint8_t contents [sizeof(Entry)]; // use to inspect raw data
int length; // probably not needed
};
};
} Node;
With Node* new_file(char* name), Node is not yet defined. Code needs typedef struct Node Node; or the like.
A big challenge to this task is the many potential errors that could creep in: file name too long, memory allocation failure, fopen open failure, read failure, file too long, ...
int length; should be size_t length; as files may be longer than INT_MAX, yet fit in an allocation.
new_file(char* name) looks like it should read a file. Let's make a helper function as there are various degrees of robustness needed. The below commonly "works" yet is technical UB (seeking to the end of a binary file). Adjust as needed.
Illustrative, untested code:
// The below does _not_ return a null character appended string,
// just the data in the file.
void *alloc_file(const char *file_name, size_t *size) {
*size = 0;
FILE *binary_stream = fopen(file_name, "rb");
if (binary_stream == NULL) {
return NULL;
}
long offset = fseek(binary_stream, SEEK_END);
rewind(binary_stream);
if (offset == -1 || (unsigned long) offset > SIZE_MAX) {
fclose(binary_stream); // Trouble or file too long
return NULL;
}
size_t file_size = (size_t) offset;
void *buf = malloc(file_size) {
fclose(binary_stream);
return NULL;
}
if (fread(buf, 1, file_size, binary_stream) != file_szie) {
fclose(binary_stream);
free(buf);
return NULL;
}
*size = file_size;
fclose(binary_stream);
return buf;
}
Now new_file is easier.
// Better as: Node* new_file(const char* name)
Node* new_file(char* name) {
if (strlen(name) >= MAX_NAME_LEN) {
return NULL // name too long
}
// Allocate to the size of the referenced object, not type.
Node* ptr_file = malloc(sizeof ptr_file[0]);
if (ptr_file == NULL) {
return NULL;
}
strcpy(ptr_file->name, name);
ptr_file->type = NodeType_file; // Some constant (OP has not shown yet).
size_t size = 0;
ptr_file->file.contents = alloc_file(name, &size);
if (ptr_file->file.contents == NULL || size > INT_MAX) {
free(ptr_file->file.contents);
free(ptr_file);
return NULL;
}
ptr_file->length = (int) size;
return ptr_file;
}
I have searched around on the internet and there were few document about how we use Ring Buffer to read from file or write to a file. So i wrote my own implementation here:
my ringbuffer structure
typedef struct Ringbuffer {
char* buffer;
int length;
int start;
int end;
int fd;
int looped;
} RingBuffer;
//to update the end of the buffer after writing
#define RingBuffer_commit_write(B, A) (B)->end += A;if((B)->end >=(B)->length){(B)->looped =1;(B)->end %= (B)->length;}
#define RingBuffer_ends_at(B) ((B)->buffer + (B)->end)
static inline int RingBuffer_available_space(RingBuffer *buffer) {
return buffer->length- RingBuffer_available_data(buffer);
}
code snippet to read from file
void read_some(int sockfd,RingBuffer *buffer) {
int byte_recv =0;
check((RingBuffer_available_space(buffer) > 0),"out of space, release some data");
if(buffer->looped == 0){
byte_recv = recv(sockfd,buffer,buffer->length - buffer->end,0);
if(byte_recv < buffer->length- buffer->end){
RingBuffer_commit_write(buffer,byte_recv);
return ;
}else{
RingBuffer_commit_write(buffer,byte_recv);
byte_recv = recv(sockfd,buffer,RingBuffer_ends_at(buffer),RingBuffer_available_space(buffer),0);
RingBuffer_commit_write(buffer,byte_recv);
}
}else{
byte_recv = recv(sockfd,buffer,RingBuffer_available_space(),0);
RingBuffer_commit_write(buffer,byte_recv)
}
error:
return;
}
This, however, is not efficient and overcomplicated in my opinion. Is there a better implementation of Ring buffer to read from file( since both socket and file are quite the same)? Any help,even an idea only,would be appreciated.
In C, I often want to handle data read from a file and data read from an array of strings the same way. Usually reading from a file is for production and from strings is for testing. I wind up writing a lot of code like this:
void handle_line(char *line, Things *things) {
...
}
Things *read_from_chars(char *lines[]) {
Things *things = Things_new();
for (int i = 0; lines[i] != NULL; i++) {
handle_line(lines[i], things);
}
return things;
}
Things *read_from_input(FILE *input) {
char *line = NULL;
size_t linelen = 0;
Things *things = Things_new();
while (getline(&line, &linelen, input) > 0) {
handle_line(line, things);
}
return things;
}
This is a duplication of effort.
Is there a way I can make an array of strings masquerade as a FILE * pointer? Or vice-versa? Or is there a better pattern for dealing with this problem?
For bonus points: the solution should make char * or char ** usable with the standard file functions like fgets and getline.
You could use a discriminated union that contains a FILE* and a pointer to the array, then write a get_next function that does the right thing with it.
typedef struct {
enum { is_file, is_array } type;
union {
FILE *file;
struct {
int index;
int size;
char **lines;
} array;
} data;
} file_or_array;
char *get_next(file_or_array foa) {
if (foa.type == is_file) {
char *line = NULL;
size_t linelen = 0;
getline(&line, &linelen, foa.data.file);
return line;
} else {
if (foa.data.array.index < foa.data.array.size) {
return strdup(foa.data.array.lines[foa.data.array.index++]);
} else {
return NULL;
}
}
}
The call to strdup() is necessary to make this work consistently. Since getline() returns a newly-allocated string, which the caller needs to free, it also does the same thing when returning a string from the array. Then the caller can safely free it in both cases.
There's a nonstandard function fmemopen that lets you open a char[] for reading or writing. It's available in most versions of GNU libc, I think, and most versions of Linux.
(This lets you read from or write to a single string, not the array of strings you asked about.)
One of the most powerful ways to handle this is via streams. I use them to hide file/string/serial ports etc
I have rolled my own stream library which I mainly use on embedded systems
the general idea is :-
typedef struct stream_s stream_t;
struct stream_s
{
BOOL (*write_n)(stream_t* stream, char* s, WORD n);
BOOL (*write_byte)(stream_t* stream, BYTE b);
BOOL (*can_write)(stream_t* stream);
BOOL (*can_read)(stream_t* stream);
BYTE (*read_byte)(stream_t* stream);
void* context;
};
then you make a whole bunch of functions
BOOL stream_create(stream_t* stream);
BOOL stream_write_n(stream_t* stream, char* s, WORD n);
BOOL stream_can_read(stream_t* stream);
BYTE stream_read_byte(stream_t* stream);
etc
that use those base function call backs.
the context in the stream struct you use to point to a struct for serial, string, file, or whatever you want. Then you have things like file_create_stream(stream_t* stream, char* filename) which will populate the callbacks on stream with the file related functions. Then for strings you have something similar but handles strings
There's more than one way to skin this particular cat, but in general the solution to this is hiding the implementation of the public interface behind an indirection which allows you to inject separate 'implementations'.
(This incarnation of your problem is also closely related to somewhat different problem of ensuring ABI compatibility between versions of code.)
To solve this in C you can do it similar to the pimpl with-inheritance in C++ (protected instead of private d-pointer, with overridden protected constructors):
You create an opaque 'reader'/'stream' object (pointer to forward declared struct w/ typedef in C) and suitably named constructor functions to instantiate the opaque object which inject the desired implementation.
Let's sketch out example header files to give you an idea of how the functions fit together. Let's start with the guts, the definition of the d-pointer/p-impl objects (N.B.: I'm omitting some boilerplate like header guards):
reader-private.h:
/* probably should be in its proper C file, but here for clarification */
struct FileReaderPrivateData {
FILE * fp;
};
/* probably should be in its proper C file, but here for clarification */
struct StringReaderPrivateData {
size_t nlines;
size_t cursor;
char ** lines;
};
/* in C we don't have inheritance, but we can 'fix' it using callbacks */
struct ReaderPrivate {
int (* close)(void* pData); /* impl callback */
ssize_t (* readLine)(void* pData, char** into); /* impl callback */
/* impl-specific data object, callbacks can type cast safely */
void * data;
};
/* works like a plain p-impl/d-pointer, delegates to the callbacks */
struct Reader {
struct ReaderPrivate * dPtr;
}
reader.h:
typedef struct Reader* Reader;
/* N.B.: buf would be a pointer to set to a newly allocated line buffer. */
ssize_t readLine(Reader r, char ** buf);
int close(Reader r);
file-reader.h
#include "reader.h"
Reader createFileReader(FILE * fp);
Reader createFileReader(const char* path);
string-reader.h
#include "reader.h"
Reader createStringReader(const char**, size_t nlines);
That's a general pattern for doing pimpl/d-pointer with inheritance in C, so you can abstract the implementation guts behind a public interface which is accessed through opaque pointers. This mechanism is generally useful to guarantee API and ABI compatibility between various implementations of the public interface and to implement a simple inheritance pattern.
Here's an implementation using fcookieopen [IIRC, BSD has something similar]:
// control for string list
struct cookie {
char **cook_list; // list of strings
int cook_maxcount; // maximum number of strings
int cook_curidx; // current index into cook_list
int cook_curoff; // current offset within item
};
int cookie_close(void *vp);
ssize_t cookie_read(void *vp,char *buf,size_t size);
cookie_io_functions_t cook_funcs = {
.read = cookie_open;
.close = cookie_close;
};
// cookie_open -- open stream
FILE *
cookie_open(char **strlist,int count,const char *mode)
// strlist -- list of strings
// count -- number of elements in strlist
// mode -- file open mode
{
cookie *cook;
FILE *stream;
cook = calloc(1,sizeof(cookie));
cook->cook_list = strlist;
cook->cook_maxcount = count;
stream = fopencookie(cook,mode,&cook_funcs);
return stream;
}
// cookie_close -- close stream
int
cookie_close(void *vp)
{
free(vp);
return 0;
}
// cookie_read -- read stream
ssize_t
cookie_read(void *vp,char *buf,size_t size)
{
cookie *cook = vp;
char *base;
ssize_t totcnt;
totcnt = 0;
while (size > 0) {
// bug out if all strings exhausted
if (cook->cook_curidx >= cook->cook_maxcount)
break;
base = cook->cook_list[cook->cook_curidx];
base += cook->cook_curoff;
// if at end of current string, start on the next one
if (*base == 0) {
cook->cook_curidx += 1;
cook->cook_curoff = 0;
continue;
}
// store character and bump buffer and count
*buf++ = *base;
size -= 1;
totcnt += 1;
cook->cook_curoff += 1;
}
return totcnt;
}
If you need this functionality just for debugging, write a fopen_strings(char *list[]) function to:
create a temporary file
open that with fopen with mode "r+"
write all your strings into it
delete the file (the FILE* can still operate on it, until it is closed either explicitly or implicitly at program end. You might need to skip this step on some operating systems that prevent deletion of open files.
rewind the stream
return the stream and let your program use it as it would a regular file.
is there a better pattern for dealing with this problem?
My proposed solution is to do function overloading.
Provide all possible parameters:
Things* readThings(FILE *f, char *l[])
{
char *line = NULL;
size_t linelen = 0;
Things *things = Things_new();
if (f)
{
while(getline(&line, &linelen, input) > 0)
handle_line(line, things);
}
else
{
for(int i = 0; lines[i] != NULL; i++)
handle_line(lines[i], things);
}
return things;
}
Things* readThingsChar(char *l[]){ return readThings(0, l); }
Things* readThingsFile(FILE *f){ return readThings(f, 0); }
How to use
FILE *f;
char *l[100];
..
Things *a = readThings(f,0); // or readThingsFile(f)
Things *b = readThings(0,l); // or readThingsChar(l)
You could embed it in the data:
Things* readThings(char *l[])
{
char *line = NULL;
size_t linelen = 0;
Things *things = Things_new();
FILE *f = NULL;
if (l[0][0]==UNIQUE_IDENTIFIER)
{
f = fopen(l[0]+1);
while(getline(&line, &linelen, input) > 0)
handle_line(line, things);
fclose(f);
}
else
{
for(int i = 0; lines[i] != NULL; i++)
handle_line(lines[i], things);
}
return things;
}
How to use
char *f[1] = { "_file.txt" };
char *l[100] = { "first line", .. "last line" };
f[0][0] = UNIQUE_IDENTIFIER;
Things *a = readThings(f);
Things *b = readThings(l);
The system call write it's defined as follow:
SYSCALL_DEFINE3(write, unsigned int, fd, const char __user *, buf, size_t, count)
{
struct file *file;
ssize_t ret = -EBADF;
int fput_needed;
file = fget_light(fd, &fput_needed);
if (file) {
loff_t pos = file_pos_read(file);
ret = vfs_write(file, buf, count, &pos);
file_pos_write(file, pos);
fput_light(file, fput_needed);
}
return ret;
}
I'd like to copy the variable buf to modify your content and
then use this new variable at:
vfs_write(file, new_buf, count, &pos);
I've tried to allocate memory to a char pointer variable with kmalloc and then I've used copy_from_user() to do the copy. Finally I've used the new variable at vfs_write(). After recompile the kernel and reboot the system I've got kernel panic error message.
Here is my implementation that generates a kernel panic error message:
SYSCALL_DEFINE3(write, unsigned int, fd, const char __user *, buf, size_t, count){
struct file *file;
ssize_t ret = -EBADF;
int fput_needed;
char *data;
data = kmalloc(count, GFP_KERNEL);
if(!data)
return ret;
copy_from_user(data, buf, count);
file = fget_light(fd, &fput_needed);
if (file) {
loff_t pos = file_pos_read(file);
ret = vfs_write(file, data, count, &pos);
file_pos_write(file, pos);
fput_light(file, fput_needed);
}
return ret;
}
How can I do this copy in kernel mode?
I'm using Linux Mint 12 - Kernel version: 3.0.30
You should probably also post your code. I.e. the changes you made to the write system call to be certain where the error is.
That said, there are checks in place that don't allow you to use kernel memory for system calls. You either need to allocate your buffer in user address space for the process (bad) or disable the checks (not as bad).
I'm not as familiar with the 3.0 kernel but this answer looks promising:
mm_segment_t old_fs;
old_fs = get_fs();
set_fs(KERNEL_DS);
/* Your syscall here */
set_fs(old_fs);
I'm trying to implement a circular buffer for use as a frame buffer with OpenCV (using C).
I've shamelessly stolen the circular buffer implementation from this post to save reinventing the wheel:
EDIT: OK, so I've redefined a few things. Namely I implemented my own circular buffer. Now I'm getting errors which don't make sense.
Here is the circular buffer implementation I'm using:
#define BUFFER_SIZE 100
typedef struct
{
IplImage* queue[BUFFER_SIZE];
IplImage *in;
IplImage *out;
int num_frames;
int in_ctr;
int out_ctr;
int update_flag;
} frame_buffer;
Here is the get function:
IplImage* buff_get()
{
IplImage* nextfr;
if(frbuff.num_frames == 0)
{
return NULL;
}
nextfr = frbuff.out++;
if(++frbuff.out_ctr == BUFFER_SIZE)
{
frbuff.out = &frbuff.queue[0];
frbuff.out_ctr = 0;
}
--frbuff.num_frames;
return nextfr;
}
Here is the put function:
int buff_put(IplImage* nextfr)
{
if(++frbuff.num_frames > BUFFER_SIZE)
{
return 0;
}
frbuff.in++;
frbuff.in = nextfr;
if(++frbuff.in_ctr == BUFFER_SIZE)
{
frbuff.in = &frbuff.queue[0];
frbuff.in_ctr = 0;
}
return 1;
}
Everything seems to go OK. Frames appear on the buffer, which I know because I can print the size out. But it all goes bad when I try to show the image that's on the buffer.
If I then try to do this:
IplImage* curr_frame = cvCreateImage(cvSize(640,480),8,3);
cvNamedWindow("proc_window",CV_WINDOW_AUTOSIZE);
cvShowImage("proc_window",curr_frame);
while(1)
{
if(buff_size() > 0)
{
if(buff_flag_check()) curr_frame = buff_get();
if(curr_frame != NULL)
{
cvShowImage("proc_window",curr_frame);
}
}
I recieve the following error upon calling cvShowImage():
OpenCV Error: Bad flag (parameter or structure field) (Unrecognized or unsupported array type) in cvGetMat, file /home/fagg/src/OpenCV-2.3.1/modules/core/src/array.cpp, line 2482
terminate called after throwing an instance of 'cv::Exception'
what(): /home/fagg/src/OpenCV-2.3.1/modules/core/src/array.cpp:2482: error: (-206) Unrecognized or unsupported array type in function cvGetMat
I'm quite confused as to what's going on here. Hopefully someone with fresher eyes than myself can see what's going on...
The snippet of code you provided won't compile as cb_init() takes a pointer to a circular_buffer as first argument.
and if you get a segfault in cb_init() it is because malloc() fails to return the requested size and the implementation in the link does not handle errors.
void cb_init(circular_buffer *cb, size_t capacity, size_t sz)
{
cb->buffer = malloc(capacity * sz);
if(cb->buffer == NULL)
// handle error
cb->buffer_end = (char *)cb->buffer + capacity * sz;//segfault when using cb->buffer which is null in case of malloc() failure
cb->capacity = capacity;
cb->count = 0;
cb->sz = sz;
cb->head = cb->buffer;
cb->tail = cb->buffer;
}