i need your help! I want to copy one file from the SD Card to the memory of my ARM Cortex A9 (to transfer it faster to the FPGA). But i dont know the start address of the file and the size. Are there any possibilities to find this information? I have some exp with FPGA, but not with mC and ARM =(
Many thanks in advance! Djrem
You're going to need some file system layer, that can interpret the contents of the SD card properly. Such cards are typically never used as raw flash, but instead use a file system layer on top which gives you directories and files. This is of course necessary when moving the card between devices, to make it interoperable.
Once you have a file system driver, you're going to be able to basically open the file on the SD card for reading, and then sit in a loop reading blocks of some suitable size. For every block read in, you simply copy copy it to the desired address in the RAM. Of course, you can read it directly to the proper address too, skipping the copy.
In pseudo-C, it would basically just be:
FILE *in;
if((in = fopen("sd0:\\file.dat", "rb")) != NULL)
{
unsigned char *target = (unsigned char *) 0xec008000; /* totally random */
size_t got;
while((got = fread(target, 1024, 1, in)) > 0)
{
target += got;
}
fclose(in);
}
Of course, you will probably not be using stdio, so the fopen(), fread() and fclose() functions will be something different depending on your file system driver.
Related
I have a bunch of huge pcap files (> 10GB) that are compressed with lzma. I need to parse them on my machine, and I do not have enough space to uncompress them first. There are many libs that can stream lzma from file. The problem is on libpcap side, I've read it's API several times, and couldn't find any way to parse a buffer. What I see in the libs' source code is that it first reads the magic byte and file header with fread:
amt_read = fread((char *)&magic, 1, sizeof(magic), fp);
...
amt_read = fread(((char *)&hdr) + sizeof hdr.magic, 1, sizeof(hdr) - sizeof(hdr.magic), fp);
And then pcap_next_packet also uses fread to read next packet from file. So it looks like it's hard to pass a buffer from lzma stream to it. On the other hand, these functions are stored in pcap_t structure as pointers. So I can implement my own procedures for it, however, this way I will have to duplicate a lot of code from libpcap. Does anybody know how to do it without hacking into libpcap?
Am I missing something in libpcap API?
Update: With #Martin and others help, I managed to make it work. I'll post the implementation, so people who look for a way to do it can use it.
if (check_file_exists("/path/to/file.pcap.xz")) {
return;
}
// first open a pipe
FILE *pipe = popen("xz -d -c /path/to/file.pcap.xz", "r");
if (!pipe) {
// handle error somehow
return;
}
char errbuff[256];
// note pcap_fopen_offline function that takes FILE* instead of name
pcap_t *pcap = pcap_fopen_offline(pipe, errbuff);
struct pcap_pkthdr *header;
uint8_t *data;
while (pcap_next_ex(pcap, &header, &data)) {
// handle packets
}
Particularly for large pcap files, it's preferable not to read the whole thing into memory first anyway. To handle the buffer management correctly, you'd need to understand the pcap format to get lengths correct, etc.
You can stream it with popen, something like:
char* cmd = asprintf("/usr/bin/xz -d -c %s", filename);
FILE* fp = popen(cmd , "r");
free(cmd);
Then read from fp just as if it was uncompressed. You can also make a wrapper function for open returning a FILE* that works out whether to pipe it through a variety of decompressors by extension or just do a plain fopen.
In general I find regular pipes preferable to named pipes where possible as it saves (a) picking a unique name and (b) cleaning them up in all error cases
Or just parse the pcap by hand, the format is fairly trivial, IIRC it's just one header struct, then one per packet.
For the class cs50, I have to read in jpeg files byte by byte from a memory card in order to look at the header information. The file compiles well, but whenever I execute the file, it returns a "segmentation fault(core dumped)" message.
Edit) Okay, now I know why I have to use an "unsigned char" instead of "int*". Can someone tell me how I can store information into files within scope for this particular code? Right now, I am trying to store information outside of an if() condition, and I don't think the fread function is actually accessing the "image" file I opened.
#include <stdio.h>
#include <string.h>
#include <math.h>
FILE * image = NULL;
int main(int argc, char* argv[])
{
FILE* infile = fopen("card.raw", "r");
if (infile == NULL)
{
printf("Could not open.\n");
fclose(infile);
return 1;
}
unsigned char storage[512];
int number = 0;
int b = floor((number) / 100);
int c = floor(((number) - (b * 100))/ 10);
int d = floor(((number) - (b * 100) - (c * 10)));
int writing = 0;
char string[5];
char* extension = ".jpg";
while (fread(&storage, sizeof(storage), 1, infile))
{
if (storage == NULL)
{
break;
}
if (storage[0] == 0xff && storage[1] == 0xd8 && storage[2] == 0xff)
{
if (storage[3] == 0xe0 || storage[3] == 0xe1)
{
if (image != NULL)
{
fclose(image);
}
sprintf(string, "%d%d%d%s", b, c, d, extension);
image = fopen(string, "w");
number++;
writing = 1;
if (writing == 1 && storage != NULL)
{
fwrite(storage, sizeof(storage), 1, image);
}
}
}
if (writing == 1 && storage != NULL)
{
fwrite(storage, sizeof(storage), 1, image);
}
if (storage == NULL)
{
fclose(image);
}
}
fclose(image);
fclose(infile);
return 0;
}
This is the problem set just in case my explanation is not clear.
recover
In anticipation of this problem set, I spent the past several days snapping photos of people I know, all of which were saved by my
digital camera as JPEGs on a 1GB CompactFlash (CF) card. (It’s
possible I actually spent the past several days on Facebook instead.)
Unfortunately, I’m not very good with computers, and I somehow deleted
them all! Thankfully, in the computer world, "deleted" tends not to
mean "deleted" so much as "forgotten." My computer insists that the CF
card is now blank, but I’m pretty sure it’s lying to me.
Write in ~/Dropbox/pset4/jpg/recover.c a program that recovers these photos.
Ummm.
Okay, here’s the thing. Even though JPEGs are more complicated than BMPs, JPEGs have "signatures," patterns of bytes that distinguish
them from other file formats. In fact, most JPEGs begin with one of
two sequences of bytes. Specifically, the first four bytes of most
JPEGs are either
0xff 0xd8 0xff 0xe0 or 0xff 0xd8 0xff 0xe1
from first byte to fourth byte, left to right. Odds are, if you find one of these patterns of bytes on a disk known to store photos
(e.g., my CF card), they demark the start of a JPEG. (To be sure, you
might encounter these patterns on some disk purely by chance, so data
recovery isn’t an exact science.)
Fortunately, digital cameras tend to store photographs contiguously on CF cards, whereby each photo is stored immediately
after the previously taken photo. Accordingly, the start of a JPEG
usually demarks the end of another. However, digital cameras generally
initialize CF cards with a FAT file system whose "block size" is 512
bytes (B). The implication is that these cameras only write to those
cards in units of 512 B. A photo that’s 1 MB (i.e.,
1,048,576 B) thus takes up 1048576 ÷ 512 = 2048 "blocks" on a CF card. But so does a photo that’s, say, one byte smaller (i.e.,
1,048,575 B)! The wasted space on disk is called "slack space."
Forensic investigators often look at slack space for remnants of
suspicious data.
The implication of all these details is that you, the investigator, can probably write a program that iterates over a copy
of my CF card, looking for JPEGs' signatures. Each time you find a
signature, you can open a new file for writing and start filling that
file with bytes from my CF card, closing that file only once you
encounter another signature. Moreover, rather than read my CF card’s
bytes one at a time, you can read 512 of them at a time into a buffer
for efficiency’s sake. Thanks to FAT, you can trust that JPEGs'
signatures will be "block-aligned." That is, you need only look for
those signatures in a block’s first four bytes.
Realize, of course, that JPEGs can span contiguous blocks. Otherwise, no JPEG could be larger than 512 B. But the last byte of a
JPEG might not fall at the very end of a block. Recall the possibility
of slack space. But not to worry. Because this CF card was brand- new
when I started snapping photos, odds are it’d been "zeroed" (i.e.,
filled with 0s) by the manufacturer, in which case any slack space
will be filled with 0s. It’s okay if those trailing 0s end up in the
JPEGs you recover; they should still be viewable.
Now, I only have one CF card, but there are a whole lot of you! And so I’ve gone ahead and created a "forensic image" of the card,
storing its contents, byte after byte, in a file called card.raw . So
that you don’t waste time iterating over millions of 0s unnecessarily,
I’ve only imaged the first few megabytes of the CF card. But you
should ultimately find that the image contains 16 JPEGs. As usual, you
can open the file programmatically with
fopen , as in the below. FILE* file = fopen("card.raw", "r");
Notice, incidentally, that ~/Dropbox/pset4/jpg contains only recover.c, but it’s devoid of any code. (We leave it to you to decide
how to implement and compile recover!) For simplicity, you should
hard-code "card.raw" in your program; your program need not accept any
command-line arguments. When executed, though, your program should
recover every one of the JPEGs from card.raw, storing each as a
separate file in your current working directory. Your program should
number the files it
outputs by naming each , ###.jpg where ### is three-digit decimal number from 000 on up. (Befriend sprintf.) You need not try to
recover the JPEGs' original names. To
check whether the JPEGs your program spit out are correct, simply double-click and take a look! If each photo appears intact,
your operation was likely a success!
Odds are, though, the JPEGs that the first draft of your code spits out won’t be correct. (If you open them up and don’t see
anything, they’re probably not correct!) Execute the command below to
delete all JPEGs in your current working directory.
rm *.jpg
If you’d rather not be prompted to confirm each deletion, execute the command below instead.
rm -f *.jpg
Just be careful with that -f switch, as it "forces" deletion without prompting you.
int* storage[512];
You define a pointer to a memory location for 512 ints, but you don't actually reserve the space (only the pointer.
I suspect you just want
int storage[512];
After this, storage is still a pointer, but now it actually points to 512 ints. Though I still think you don't want this. You need 'bytes' not ints. The nearest C has are unsigned char. So the final declaration is:
unsigned char storage[512];
Why? Because read reads into consecutive bytes. If you read into ints, then you will read 4 bytes into each int (because an int occupies 4 bytes).
There are a number of problems in your program. The first is that you have not opened the file in binary mode.
The second is that you are doing unnecessary pointer arithmetic. Why not—
char buffer [BUFFERSIZE] ;
....
if (buffer [ii] == WHATEVER)
I created a C program which will run some system() commands and save the output to a .txt file in a particular folder of C drive (program is to be run on domain clients with startup privileges), from where I'll filter the data and show it on output console screen.
Everything worked fine, but I couldn't design it for those PCs who have OS installed in other drives (e.g. D:, E:, etc), since they will not have this particular folder in their C: drive. I can't write temporary .txt files anywhere else due to group policies.
Is there any method to stream this data directly into any array variable? I went through popen() function, but it would require a very large array of unpredicted size to be defined (since the output of system() command may be very large). For example, exporting the registry keys of HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\uninstall into a .txt file. Its size may be up to 50KB or bigger.
reg export HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\uninstall C:\\..(somewhere)....\\reg_output.txt
There are some more similar commands with large output. I don't know Win API programming yet, thus I am using system command. Can there be an easy alternative to writing to .txt ?
#Megharaj!
As you used
char line[100];
fp = popen("ifconfig eth0", "r");
fgets(line, 100, fp);
I'll have to use
char reg_output[100000];
fp=popen("reg export HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\uninstall ????","r");
fgets(line,100000,fp);
Where do I export the registry values? (Since the DOS command for exporting this needs to write it to a file), as compared to following code I am using.
Assigning a space of 100000 isn't sure that it will not be error prone. And assigning too high value also will affect the memory on startup (I'm not sure but guess so).
While using file handling I do it as:
char line[5000]; FILE* fp_reg; int ch,n=0;
system("reg export HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\uninstall c:\\registries.txt");
fp_reg=fopen("c:\\registries.txt","r");
while((ch=fgetc(fp_reg))!=EOF)
{
if(isalnum(ch) || ch=='\n') //allow only alpha numeric & '\n' to pass to array
{ line[n]=ch;
if(ch=='\n')
{ filter_for_software(line,n); //Send the array for filtering function
n=0; // Reset the array 'line'
}
n++;
}
}
(I didn't copy the code from source, since I am sitting on a different PC. But the code is almost similar to this. Some errors may come if you copy this code & run.)
This way, I take each line from file and send it for filtering to get 'installed software's name'. How do I do it with 'popen()', so that I could pick up a line and throw it onto a function. Plz write some code also for this.
If you want to store the output in an array, you will have to allocate an array of appropriate size. There is just no way around that.
But you don't need to do that if you use popen. You get a FILE * handle and can just read the output in small parts and process it on the go.
from where I'll filter the data and show it on output console screen.
How do you do that? With the shell or in C? If in C, then you read the output of the systemed command just like you would read your txt file. The only difference is that you need to close it with pclose instead of fclose, so there are only advantages to doing it with a text file.
If in shell, then you can start the program you use with another call to popen, this time in a "w" (write) direction and write the output of one pipe as input to the other. However, in this case you could have just called a shell with an anonymous pipe in the first place. ;-)
Edit:
Your own answer makes clear that your requirements are quite different from what you think they are. There is no way to read the output of reg export back via a pipe and popen if it insists on writing to a file.
What you can try is to write a file to your temp folder; you should be allowed to create files there, otherwise Windows will not work correctly. Just specify something like "%TEMP%\reg.out" as file and read your data back from there.
If that doesn't work, you are out of luck with reg export. But you can use some Windows API function for querying the registry directly. Here is a starting point.
Alternatively, you might want to look into the possibility of employing PowerShell. This question might be of interest to you.
In linux to use the values from the system command i use popen, I am just giving an example of code that I had written some time long back, to get the ip address of the pc by system command "ifconfig eth0" to the string/a file. see the example
void get_my_ip(char *ip_mac_address)
{
FILE *fp,*output;
char *start=NULL;
char *end=NULL;
char line[100];
output=fopen("my_ip_address.txt", "w");
if(output == NULL) {
printf("error creating outputfile\n");
return -1;
}
printf("program to self query the ip address\n");
fp = popen("ifconfig eth0", "r");
fgets(line, 100, fp);
start=strstr(line, CHECK_STRING_MAC);
start = start + 7;
fwrite(start, 1, 17, output); start = NULL;
fgets(line, 100, fp);
start=strstr(line, CHECK_STRING_IP);
start = start + 10;
fwrite(start, 1, 14, output);
fclose(output);
pclose(fp);
if( access("my_ip_address.txt", F_OK ) != -1 ) {
printf("found file having ip address\n");
output=fopen("my_ip_address.txt", "r");
fgets(ip_mac_address, 32, output);
}
else
printf("unabe to find file with ip address\n");
fclose(output);
printf("my ip and mac address adress is %s \n",ip_mac_address);
}
You can create a temporary file using some API from Windows, and store your data in it.
http://msdn.microsoft.com/en-us/library/windows/desktop/aa363875(v=vs.85).aspx
There is a C++ example in the link, you should be able to adapt it to your case.
So, I prepared the code to stream the output of a DOS command to a file pointer in read mode and check it line by line:
char array_sys[200]; FILE *fp_sys; int ch,n=0;
fp=popen("systeminfo","r"); //runs a DOS command & read it with a file pointer
if(fp_sys==NULL) printf("can't open file\n");
while((ch=fgetc(fp_sys))!=EOF) //READ CHARACTERS FROM FILE POINTER UNTIL FILE ENDS
{
array_sys[n]=ch; //ASSIGN CH TO EACH ELEMENT OF ARRAY
n++; //INCREMENT ELEMENTS OF ARRAY:-arr_sys
if(ch=='\n') //IF ELEMENTS MEET A NEW LINE CHARACTER
{
disp_date(array_sys,n); //PASS ARRAY TO FUNCTION:-disp_date
n=0; //RESET THE ARRAY
}
}
Now this is how I process array in function to get the name of operating system.
void disp_date(char array_sys[],int ind)
{
char os_name[9]={"OS Name:"};
if(strstr(array_sys,os_name)) //IF 'OS NAME' IS PRESENT IN ARRAY
{
printf("%s",array_sys); //PRINT THE ARRAY
}
}
I am currently working on a project that involves transferring a very large file (about 6GB) from one Linux Server to another. The servers run on Debian Squeeze.
In order to achieve my main goal, I initially send the file's name and size to the destination machine, and I create an empty file for storing the data blocks that I progressively receive from the source machine.
My problem is that the creation of a 6GB file takes too long in my server. To make it more clear I use the following C routine in order to create the new file:
void create_file(char* f_name, long long f_size) {
char* bs, *of, *s_f_size, *count;
if((pid = fork()) < 0) {
perror("fork() failed.");
return;
}
if(pid == 0) {
//Call execl
of = (char*) malloc(sizeof(char)*(strlen("of=") + strlen(f_name) + 1));
s_f_size = (char*) malloc(sizeof(char)*32);
sprintf(s_f_size, "%lld", file_size);
count = (char*) malloc(sizeof(char)*(strlen("count=") + strlen(s_f_size) + 1));
strcpy(of, "of=");
strcat(of, f_name);
strcpy(count, "count=");
strcat(count, s_f_size);
ret = execl("/bin/dd", "dd", "if=/dev/zero", of, "bs=1", count, (char*) 0);
if(ret < 0) {
perror("execl() failed");
free(s_f_size);
free(of);
free(count);
return;
}else {
free(s_f_size);
free(of);
free(count);
return;
}
}else {
status = 0;
wpid = wait(&status);
}
}
I used the Linux dd command because I thought that it would be the quickest way to create an empty 6GB file. However, it takes about 15 minutes to complete. Is there a way to create the empty file faster? What am I doing wrong?
Thank you for your time.
Sincerely,
Nick
In addition to what Joachim Pileborg suggested, you can also use posix_fallocate() to pre-allocate space for your file.
First creat the file, then lseek to the wanted end, and write a dummy byte. Very quick way to create an arbitrary large but sparse file.
If you don't want the file to be sparse, then find out the block size of the drive (can be found out using stat on most POSIX platforms). Create a buffer of that size, and write it to the file until the wanted size.
If the stat structure doesn't have the st_blksize member, then most filesystems have a blocksize of 4 or 8 kB. You can probably make this buffer larger, but not too large. Experiment and benchmark!
If you're using kernel v2.6.31+ and if filesystem supports it, consider using fallocate:
fallocate -l 6GB hugefile
It preallocates blocks to a file.
Creating large files takes long because there's a lot on the file system the OS has to do. Only in case of sparse files this can be skipped (see Joachim Pileborg's answer for that). A sparse file is a file containing "holes" (large chunks of zero bytes). Such a file does not use as much space as it is large. Creating such a file beforehand will produce the file with the correct size very fast.
In case you want to reserve the disk space to avoid running out of it before the transmission is complete, a sparse file won't do. You will have to write at least one byte into each block then to avoid the holes of a sparse file. I'm not sure this will be faster than simply dump zeros into the file until it has the desired size, as you already do.
i remember, i've used open system call to create a empty file. Then dump data to file.
In case of partial data write, keep seek the position and dump from there. If file exists use the file to overwrite the data into it.
With respect to performance this approach was quite good.
I have been given a raw file that holds several jpg images. I have to go through the file, find each jpg image, and put those images each in a separate file. So far I have code that can find each where each image begins and ends. I also have written code that names several file names I can use to put the pictures in. It is an array: char filename[] , that holds the names: image00.jpg - image29.jpg .
What I cannot figure out is how to open a file every time I find an image, an then close that file and open a new one for the next image. Do I need to use fwrite()? Also, each image is in blocks of 512 bytes, so I only have to check for a new image every 512 bytes once I find the first one. Do I need to add that into fwrite?
So, to summarize my questions, I don't understand how to use fwrite(), if that is what I should be using to write to these files.
Also, I do not know how to open the files using the names I have already created.
Thanks in advance for the help. Let me know if I need to post any other code.
Use fopen(rawfilename, "rb"); to open the raw file for reading. and fread to read from it.
Use fopen(outfilename, "wb"); to open output file for writing and fwrite to write to it.
As mentioned in my comment, you are assigning char *[] to char*, use char filename[] = "image00.jpg"; instead.
Don't forget to close each file after you finish its processing (r/w) (look at fclose() at the same site of other links)
Decide how much bytes to read each time by parsing the jpeg header. Use malloc to allocate the amount of bytes needed to be read, and remember, for each allocation of buffer you need to free the allocated buffer later.
Pretty much any book on C programming should cover the functions you need. As MByD pointed out, you'll want to use the functions fopen(), fwrite(), and fclose().
I imagine your code may include fragments that look something like
/* Warning: untested and probably out-of-order code */
...
char **filename = {
"image00.jpg", "image01.jpg", "image02.jpg",
...
"image29.jpg" };
...
int index = 0;
const int blocksize = 512; /* bytes */
...
index++;
...
FILE * output_file = fopen( filename[index], "wb");
fwrite( output_data, 1, blocksize, output_file );
fclose(output_file);
...