I have a bunch of huge pcap files (> 10GB) that are compressed with lzma. I need to parse them on my machine, and I do not have enough space to uncompress them first. There are many libs that can stream lzma from file. The problem is on libpcap side, I've read it's API several times, and couldn't find any way to parse a buffer. What I see in the libs' source code is that it first reads the magic byte and file header with fread:
amt_read = fread((char *)&magic, 1, sizeof(magic), fp);
...
amt_read = fread(((char *)&hdr) + sizeof hdr.magic, 1, sizeof(hdr) - sizeof(hdr.magic), fp);
And then pcap_next_packet also uses fread to read next packet from file. So it looks like it's hard to pass a buffer from lzma stream to it. On the other hand, these functions are stored in pcap_t structure as pointers. So I can implement my own procedures for it, however, this way I will have to duplicate a lot of code from libpcap. Does anybody know how to do it without hacking into libpcap?
Am I missing something in libpcap API?
Update: With #Martin and others help, I managed to make it work. I'll post the implementation, so people who look for a way to do it can use it.
if (check_file_exists("/path/to/file.pcap.xz")) {
return;
}
// first open a pipe
FILE *pipe = popen("xz -d -c /path/to/file.pcap.xz", "r");
if (!pipe) {
// handle error somehow
return;
}
char errbuff[256];
// note pcap_fopen_offline function that takes FILE* instead of name
pcap_t *pcap = pcap_fopen_offline(pipe, errbuff);
struct pcap_pkthdr *header;
uint8_t *data;
while (pcap_next_ex(pcap, &header, &data)) {
// handle packets
}
Particularly for large pcap files, it's preferable not to read the whole thing into memory first anyway. To handle the buffer management correctly, you'd need to understand the pcap format to get lengths correct, etc.
You can stream it with popen, something like:
char* cmd = asprintf("/usr/bin/xz -d -c %s", filename);
FILE* fp = popen(cmd , "r");
free(cmd);
Then read from fp just as if it was uncompressed. You can also make a wrapper function for open returning a FILE* that works out whether to pipe it through a variety of decompressors by extension or just do a plain fopen.
In general I find regular pipes preferable to named pipes where possible as it saves (a) picking a unique name and (b) cleaning them up in all error cases
Or just parse the pcap by hand, the format is fairly trivial, IIRC it's just one header struct, then one per packet.
Related
I'm creating a program using lex and yacc to parse text, but i need create a parser of various content. I don't wish use the stdin, if i using FILE *yyin to specify the input, i can change the source. I need can call the function from library parse (created with lex file and yacc file) to parse this content and receive a result.
/**
* This i don't know is possible, receive a char * and return a FILE*
*/
FILE *function_parse_to_file(char* text){
FILE *fp = NULL;
/**
* is really necessary create a temporary file with content text?
*/
return fp
}
/**
* I need call from other library or application
*/
char *function_parse_from_lex(char* text){
yyin = function_parse_to_file(text);
init();
yyparse();
fclose(yyin);
}
On a POSIX-2008-compliant system (and on Linux), you can use fmemopen to get a FILE* handle on an in-memory buffer.
You can define YY_INPUT macro with three arguments: buffer, result, max_size, where:
buffer - input with buffer where to read data,
result - output to store number of bytes read
max_size - input with buffer size
Just include the macro definition in your Lex file using header or inline and it will be used instead of fread(...)
You really haven't stated your question clearly, but I am going to assume you want to create a FILE * which will return the contents of the string pointed to by the char * when data is read from it. You could simply create a pipe and then invoke fdopen on the read side. It is a bit dangerous to just write the data into the write side, since the write might block and lead to a deadlock, but you can certainly fork a child and have the child write the data into the pipe.
On the other hand, there's no real reason not to create a temporary file. Assuming you are going to unlink the file after you read it, there's very little chance of the data ever going to disk (the OS will keep it in memory) If you're really concerned to can use a path on a ram disk.
Disclaimer: this is for an assignment. I am not asking for explicit code. Rather, I only ask for enough help that I may understand my problem and correct it myself.
I am attempting to recreate the Unix ar utility as per a homework assignment. The majority of this assignment deals with file IO in C, and other parts deal with system calls, etc..
In this instance, I intend to create a simple listing of all the files within the archive. I have not gotten far, as you may notice. The plan is relatively simple: read each file header from an archive file and print only the value held in ar_hdr.ar_name. The rest of the fields will be skipped over via fseek(), including the file data, until another file is reached, at which point the process begins again. If EOF is reached, the function simply terminates.
I have little experience with file IO, so I am already at a disadvantage with this assignment. I have done my best to research proper ways of achieving my goals, and I believe I have implemented them to the best of my ability. That said, there appears to be something wrong with my implementation. The data from the archive file does not seem to be read, or at least stored as a variable. Here's my code:
struct ar_hdr
{
char ar_name[16]; /* name */
char ar_date[12]; /* modification time */
char ar_uid[6]; /* user id */
char ar_gid[6]; /* group id */
char ar_mode[8]; /* octal file permissions */
char ar_size[10]; /* size in bytes */
};
void table()
{
FILE *stream;
char str[sizeof(struct ar_hdr)];
struct ar_hdr temp;
stream = fopen("archive.txt", "r");
if (stream == 0)
{
perror("error");
exit(0);
}
while (fgets(str, sizeof(str), stream) != NULL)
{
fscanf(stream, "%[^\t]", temp.ar_name);
printf("%s\n", temp.ar_name);
}
if (feof(stream))
{
// hit end of file
printf("End of file reached\n");
}
else
{
// other error interrupted the read
printf("Error: feed interrupted unexpectedly\n");
}
fclose(stream);
}
At this point, I only want to be able to read the data correctly. I will work on seeking the next file after that has been finished. I would like to reiterate my point, however, that I'm not asking for explicit code - I need to learn this stuff and having someone provide me with working code won't do that.
You've defined a char buffer named str to hold your data, but you are accessing it from a separate memory ar_hdr structure named temp. As well, you are reading binary data as a string which will break because of embedded nulls.
You need to read as binary data and either change temp to be a pointer to str or read directly into temp using something like:
ret=fread(&temp,sizeof(temp),1,stream);
(look at the doco for fread - my C is too rusty to be sure of that). Make sure you check and use the return value.
I'm trying to implement a basic file server. I have been trying to use the sendfile command found here: http://linux.die.net/man/2/sendfile I'm using TCP.
I can have it send fine, but its sending in binary and I'm not sure if thats the hang up.
I am trying to receive the file with recv, but it isn't coming through correctly.
Is there a special way to receive a binary file, and put it into a string?
EDIT:
Asked to supply some code, here it is:
SENDFILE Call (from Server process)
FILE * file = fopen(filename,"rb");
if ( file != NULL)
{
/*FILE EXISITS*/
//Get file size (which is why we opened in binary)
fseek(file, 0L, SEEK_END);
int sz = ftell(file);
fseek(file,0L,SEEK_SET);
//Send the file
sendfile(fd,(int)file,0,sz);
//Cleanup
fclose(file);
}
RECIEVE Call (from Client process, even more basic than a loop, just want a single letter)
//recieve file
char fileBuffer[1000];
recv(sockfd,fileBuffer,1,0);
fprintf(stderr,"\nContents:\n");
fprintf(stderr,"%c",fileBuffer[0]);
EDIT: wrote some code for checking return values. sendfile is giving errno 9 - bad file number. Which im assuming is at my second file descriptor in the call (the one for the file i'm sending). I cast it as an int because sendfile was complaining it wasn't an int.
How should I use send file given the file pointer code I have used above in th sendfile call?
You cannot use sendfile() with a FILE*, you need a file descriptor as given by open(), close() and friends. You cannot just cast a FILE* into an int and thinking it would work.
Maybe you should read the sendfile() manpage for more information.
There is no special way. You just receive with read() or recv().
Probably, you've got your receiving code wrong.
I have been given a raw file that holds several jpg images. I have to go through the file, find each jpg image, and put those images each in a separate file. So far I have code that can find each where each image begins and ends. I also have written code that names several file names I can use to put the pictures in. It is an array: char filename[] , that holds the names: image00.jpg - image29.jpg .
What I cannot figure out is how to open a file every time I find an image, an then close that file and open a new one for the next image. Do I need to use fwrite()? Also, each image is in blocks of 512 bytes, so I only have to check for a new image every 512 bytes once I find the first one. Do I need to add that into fwrite?
So, to summarize my questions, I don't understand how to use fwrite(), if that is what I should be using to write to these files.
Also, I do not know how to open the files using the names I have already created.
Thanks in advance for the help. Let me know if I need to post any other code.
Use fopen(rawfilename, "rb"); to open the raw file for reading. and fread to read from it.
Use fopen(outfilename, "wb"); to open output file for writing and fwrite to write to it.
As mentioned in my comment, you are assigning char *[] to char*, use char filename[] = "image00.jpg"; instead.
Don't forget to close each file after you finish its processing (r/w) (look at fclose() at the same site of other links)
Decide how much bytes to read each time by parsing the jpeg header. Use malloc to allocate the amount of bytes needed to be read, and remember, for each allocation of buffer you need to free the allocated buffer later.
Pretty much any book on C programming should cover the functions you need. As MByD pointed out, you'll want to use the functions fopen(), fwrite(), and fclose().
I imagine your code may include fragments that look something like
/* Warning: untested and probably out-of-order code */
...
char **filename = {
"image00.jpg", "image01.jpg", "image02.jpg",
...
"image29.jpg" };
...
int index = 0;
const int blocksize = 512; /* bytes */
...
index++;
...
FILE * output_file = fopen( filename[index], "wb");
fwrite( output_data, 1, blocksize, output_file );
fclose(output_file);
...
I'm looking for a way to pass in a FILE * to some function so that the function can write to it with fprintf. This is easy if I want the output to turn up in an actual file on disk, say. But what I'd like instead is to get all the output as a string (char *). The kind of API I'd like is:
/** Create a FILE object that will direct writes into an in-memory buffer. */
FILE *open_string_buffer(void);
/** Get the combined string contents of a FILE created with open_string_buffer
(result will be allocated using malloc). */
char *get_string_buffer(FILE *buf);
/* Sample usage. */
FILE *buf;
buf = open_string_buffer();
do_some_stuff(buf); /* do_some_stuff will use fprintf to write to buf */
char *str = get_string_buffer(buf);
fclose(buf);
free(str);
The glibc headers seem to indicate that a FILE can be set up with hook functions to perform the actual reading and writing. In my case I think I want the write hook to append a copy of the string to a linked list, and for there to be a get_string_buffer function that figures out the total length of the list, allocates memory for it, and then copies each item into it in the correct place.
I'm aiming for something that can be passed to a function such as do_some_stuff without that function needing to know anything other than that it's got a FILE * it can write to.
Is there an existing implementation of something like this? It seems like a useful and C-friendly thing to do -- assuming I'm right about the FILE extensibility.
If portability is not important for you, you can take a look on fmemopen and open_memstream. They are GNU extensions, hence only available on glibc systems. Although it looks like they are part of POSIX.1-2008 (fmemopen and open_memstream).
I'm not sure if it's possible to non-portably extend FILE objects, but if you are looking for something a little bit more POSIX friendly, you can use pipe and fdopen.
It's not exactly the same as having a FILE* that returns bytes from a buffer, but it certainly is a FILE* with programmatically determined contents.
int fd[2];
FILE *in_pipe;
if (pipe(fd))
{
/* TODO: handle error */
}
in_pipe = fdopen(fd[0], "r");
if (!in_pipe)
{
/* TODO: handle error */
}
From there you will want to write your buffer into fd[1] using write(). Careful with this step, though, because write() may block if the pipe's buffer is full (i.e. someone needs to read the other end), and you might get EINTR if your process gets a signal while writing. Also watch out for SIGPIPE, which happens when the other end closes the pipe. Maybe for your use you might want to do the write of the buffer in a separate thread to avoid blocking and make sure you handle SIGPIPE.
Of course, this won't create a seekable FILE*...
I'm not sure I understand why you want to mess up with FILE *. Couldn't you simply write to a file and then load it in string?
char *get_file_in_buf(char *filename) {
char *buffer;
... get file size with fseek or fstat ...
... allocate buffer ...
... read buffer from file ...
return buffer;
}
If you only want to "write" formatted text into a string, another option could be to handle an extensible buffer using snprintf() (see the answers to this SO question for a suggestion on how to handle this: Resuming [vf]?nprintf after reaching the limit).
If, instead, you want to create a type that can be passed transparently to any function taking a FILE * to make them act on string buffers, it's a much more complex matter ...