Currently I am trying to read a binary file 16 bytes at a time. This is great until the end of the file where it reads a full 16 bytes regardless of whether the file has ended or not. I have a file with "ABCDEFGHIJKLMNOPQRSTUV" written into it and it reads "QRSTUVGHIJKLMNOP" on the second pass after successfully receiving the first 16. How can I stop it from re reading those bytes? This is what I have currently. It obviously does not print the second pass of data because fread does not return 16. I should note I MUST read 16 bytes at a time.
#define BS 16
unsigned char buffer[BS]; // for storing each 16 bytes read
int i = 0; // for iterating through readbytes
while (fread(buffer, 1, sizeof(buffer), ifp) == BS) {
while (i < BS) {
printf("Read: %c\n",buffer[i]);
i++;
}
i = 0;
}
It doesn't reread the bytes GHI…NOP; they are still in the buffer from the previous successful read. When fread() reports a short read, the contents of the buffer after the data it reports reading are indeterminate. You need to save the return value from fread() — it's a size_t — and use that to guide you. It reports 6 bytes; accessing anything after that leads to undefined (unspecified, indeterminate — generically 'undesirable') behaviour.
You should define int i = 0; inside the outer while loop; then you wouldn't need the trailing i = 0; after the inner loop. Or even just a routine for loop, as M.M commented.
Hence:
#define BS 16
unsigned char buffer[BS];
size_t nbytes;
while ((nbytes = fread(buffer, 1, sizeof(buffer), ifp)) == BS)
{
for (size_t i = 0; i < BS; i++)
printf("Read: %c\n", buffer[i]);
}
printf("Short read (%zu bytes):\n", nbytes);
for (size_t i = 0; i < nbytes; i++)
printf("Short: %c\n", buffer[i]);
Related
I have been doing some practice problems for job interviews and I came across a function that I can't wrap my mind on how to tackle it. The idea is to create a function that takes the name of two files, and the allowed buffer size to read from file1 and allowed buffer size for write to file2. if the buffer size is the same, I know how to go trough the question, but I am having problems figuring how to move data between the buffers when the sizes are of different. Part of the constraints is that we have to always fill the write buffer before writing it to file. if file1 is not a multiple of file2, we pad the last buffer transfer with zeros.
// input: name of two files made for copy, and their limited buffer sizes
// output: number of bytes copied
int fileCopy(char* file1,char* file2, int bufferSize1, int bufferSize2){
int bytesTransfered=0;
int bytesMoved=o;
char* buffer1, *buffer2;
FILE *fp1, *fp2;
fp1 = fopen(file1, "r");
if (fp1 == NULL) {
printf ("Not able to open this file");
return -1;
}
fp2 = fopen(file2, "w");
if (fp2 == NULL) {
printf ("Not able to open this file");
fclose(fp1);
return -1;
}
buffer1 = (char*) malloc (sizeof(char)*bufferSize1);
if (buffer1 == NULL) {
printf ("Memory error");
return -1;
}
buffer2 = (char*) malloc (sizeof(char)*bufferSize2);
if (buffer2 == NULL) {
printf ("Memory error");
return -1;
}
bytesMoved=fread(buffer1, sizeof(buffer1),1,fp1);
//TODO: Fill buffer2 with maximum amount, either when buffer1 <= buffer2 or buffer1 > buffer2
//How do I iterate trough file1 and ensuring to always fill buffer 2 before writing?
bytesTransfered+=fwrite(buffer2, sizeof(buffer2),1,fp2);
fclose(fp1);
fclose(fp2);
return bytesTransfered;
}
How should I write the while loop for the buffer transfers before the fwrites?
I am having problems figuring how to move data between the buffers when the sizes are of different
Layout a plan. For "some practice problems for job interviews", a good plan and ability to justify it is important. Coding, although important, is secondary.
given valid: 2 FILE *, 2 buffers and their sizes
while write active && read active
while write buffer not full && reading active
if read buffer empty
read
update read active
append min(read buffer length, write buffer available space) of read to write buffer
if write buffer not empty
pad write buffer
write
update write active
return file status
Now code it. A more robust solution would use a struct to group the FILE*, buffer, size, offset, length, active variables.
// Return true on problem
static bool rw(FILE *in_s, void *in_buf, size_t in_sz, FILE *out_s,
void *out_buf, size_t out_sz) {
size_t in_offset = 0;
size_t in_length = 0;
bool in_active = true;
size_t out_length = 0;
bool out_active = true;
while (in_active && out_active) {
// While room for more data
while (out_length < out_sz && in_active) {
if (in_length == 0) {
in_offset = 0;
in_length = fread(in_buf, in_sz, 1, in_s);
in_active = in_length > 0;
}
// Append a portion of `in` to `out`
size_t chunk = min(in_length, out_sz - out_length);
memcpy((char*) out_buf + out_length, (char*) in_buf + in_offset, chunk);
out_length += chunk;
in_length -= chunk;
in_offset += chunk;
}
if (out_length > 0) {
// Padding only occurs, maybe, on last write
memset((char*) out_buf + out_length, 0, out_sz - out_length);
out_active = fwrite(out_buf, out_sz, 1, out_s) == out_sz;
out_length = 0;
}
}
return ferror(in_s) || ferror(out_s);
}
Other notes;
Casting malloc() results not needed. #Gerhardh
// buffer1 = (char*) malloc (sizeof(char)*bufferSize1);
buffer1 = malloc (sizeof *buffer1 * bufferSize1);
Use stderr for error messages. #Jonathan Leffler
Open the file in binary.
size_t is more robust for array/buffer sizes than int.
Consider sizeof buffer1 vs. sizeof (buffer1) as parens not needed with sizeof object
while(bytesMoved > 0) {
for(i=0; i<bytesMoved && i<bufferSize2; i++)
buffer2[i]=buffer1[i];
bytesTransfered+=fwrite(buffer2, i,1,fp2);
bytesMoved-=i;
}
If bufferSize1 is smaller than the filesize you need an outer loop.
As the comments to your question have indicated, this solution is not the best way to transfer data from 1 file to another file. However, your case has certain restrictions, which this solution accounts for.
(1) Since you are using a buffer, you do not need to read and write 1 char at a time, but instead you can make as few calls to those functions possible.
size_t fread(void *ptr, size_t size, size_t nmemb, FILE *stream);
:from the man page for fread, nmemb can = bufferSize1
(2) You will need to check the return from fread() (i.e. bytesMoved) and compare it with both of the bufferSize 1 and 2. If (a) bytesMoved (i.e. return from fread()) is equal to bufferSize1 or if (b) bufferSize2 is less than bufferSize1 or the return from fread(), then you know that there is still data that needs to be read (or written). So, therefore you should begin the next transfer of data, and when completed return to the previous step you left off on.
Note: The pointer to the File Stream in fread() and fwrite() will begin where it left off in the event that the data is larger than the bufferSizes.
PseudoCode:
/* in while() loop continue reading from file 1 until nothing is left to read */
while (bytesMoved = fread(buffer1, sizeof(buffer1), bufferSize1, fp1))
{
/* transfer from buffer1 to buffer2 */
for(i = 0; i < bytesMoved && i < bufferSize2; i++)
buffer2[i] = buffer1[i];
buffer2[i] = '\0';
iterations = 1; /* this is just in case your buffer2 is super tiny and cannot store all from buffer1 */
/* in while() loop continue writing to file 2 until nothing is left to write
to upgrade use strlen(buffer2) instead of bufferSize2 */
while (bytesTransfered = fwrite(buffer2, sizeof(buffer2), bufferSize2, fp2))
{
/* reset buffer2 & write again from buffer1 to buffer2 */
for(i = bufferSize2 * iterations, j = 0; i < bytesMoved && j < bufferSize2; i++, j++)
buffer2[j] = buffer1[i];
buffer2[j] = '\0';
iterations++;
}
/* mem reset buffer1 to prepare for next data transfer*/
}
According to the man page of fwrite:
fread() and fwrite() return the number of items successfully read or written (i.e., not the number
of characters). If an error occurs, or the end-of-file is reached, the return value is a short
item count (or zero).
I'm wondering if it possible for fwrite(ptr, 65537, 1, fp) to write only 512 bytes and yet return zero?
I tried to get a quick source trace to glibc-2.12.2 source code.
malloc/malloc.c:#define fwrite(buf, size, count, fp) _IO_fwrite (buf, size, count, fp)
I guess the real implementation is in the iofwrite.c.
_IO_size_t
_IO_fwrite (buf, size, count, fp)
const void *buf;
_IO_size_t size;
_IO_size_t count;
_IO_FILE *fp;
{
_IO_size_t request = size * count;
_IO_size_t written = 0;
CHECK_FILE (fp, 0);
if (request == 0)
return 0;
_IO_acquire_lock (fp);
if (_IO_vtable_offset (fp) != 0 || _IO_fwide (fp, -1) == -1)
written = _IO_sputn (fp, (const char *) buf, request);
_IO_release_lock (fp);
/* We have written all of the input in case the return value indicates
this or EOF is returned. The latter is a special case where we
simply did not manage to flush the buffer. But the data is in the
buffer and therefore written as far as fwrite is concerned. */
if (written == request || written == EOF)
return count;
else
return written / size;
}
So, I guess it is possible return 0 in the return code of fwrite even if it has really written some bytes into the file.
Yes — it is possible for fwrite(ptr, 65537, 1, fp) to return 0 as the return code even if it really wrote some bytes into the file.
For example, if there is only space left on the disk for 512 bytes, then you have a short write, but it wrote 0 complete units of 65537 bytes (that's a curious number; usually it would be 65536, wouldn't it?), so the return value must be 0.
If you want to know how much was really written, you should use:
nbytes = fwrite(ptr, 1, 65537, fp);
to get the exact byte count; now you'd get 512 back from your hypothetical short write.
I am currently trying to write a small program that tests a smaller functionality of a udp socket program. This section of code opens a file, stores the contents of the file into a buffer and then writes every 1024 bytes of the buffer into a section of another buffer (which is going to be sent as packets in the udp application), I also store a sequence number at the beginning of each packet. The file_buffer gets correctly filled but the packet_buffer does not. (at packet_buffer[0], it is all null)
The problem is that this works for all cases except for the 0th place in the buffer. Here everything is null.
My code is below, I have been stumped for hours trying to debug this little error.
Any tips, tricks, solutions?
#include "sendto_.h"
#define MAXBUFSIZE 102400
#define WINDOWSIZE 6
#define THEWINDOW 100
int main (int argc, char * argv[])
{
/** file pointer to point to file opened by client to be */
FILE* fp; /**sent to the server*/
/** Identifiers for the start and the*/
int window_start, window_end; /** end of the window */
/** window for GBN sliding window */
char window[THEWINDOW];
/** buffer to hold the contents of the file to be sent to server */
char *file_buffer;
/*set of buffers to be used to send approprate window packets */
char packet_buffer[100][1024];
int sequence_id = 0;
int i, j, size, read_len;
if(!(fp = fopen(argv[1], "r")))
fprintf(stderr, "Error opening '%s': %s\n", argv[1], strerror(errno));
// obtain file size:
fseek (fp , 0 , SEEK_END);
size = ftell (fp);
rewind (fp);
file_buffer = (char *) malloc(size * sizeof(char));
/** size */
read_len = fread(file_buffer, 1, size, fp);
if(read_len > MAXBUFSIZE){
printf("file size is too large: %d\n Exiting safely\n", read_len);
free(file_buffer);
return 1;
}
//printf("file buffer: %s", file_buffer);
//snprintf(size_buffer, 10, "%d", read_len);
/******************
sendto_() sends immediately.
it will report an error if the message fails to leave the computer
however, with UDP, there is no error if the message is lost in the network once it leaves the computer.
******************/
//bzero(buffer,sizeof(buffer));
/*loop that is going to deal with the main functionality of this
program: sliding window, sending packets and receiving ACKS.*/
int k = 0;
int count = 0;
int temp;
/*setting up packets to be sent */
for(i = 0; i < THEWINDOW;i++){
temp = i;
packet_buffer[i][0] = i;
sequence_id = packet_buffer[i][0];
printf("sequence # of packet = %d\n", sequence_id);
for(j = 0; j < 1025; j++){
packet_buffer[i][j+1] = file_buffer[(j)+k];
count += 1;
if(count == size){
j = 1026;
i = THEWINDOW;
}
}
printf("count: %d\n", count);
printf("Wrote section of file into array: %s\n", packet_buffer[temp]);
k += 1025;
//read info into each set of the packet buffer
}
window_start = 0;
window_end = 5;
return 0;
}
I would think:
char packet_buffer[100][1024 + 1];
...
for(i = 0, count=0; i < THEWINDOW && count < size;i++){
packet_buffer[i][0] = i;
sequence_id = packet_buffer[i][0];
printf("sequence # of packet = %d\n", sequence_id);
for(j = 1; j < 1025 && count < size; ){
packet_buffer[i][j++] = file_buffer[count++];
}
printf("count: %d\n", count);
printf("Wrote section of file into array: %s\n", packet_buffer[i]);
//read info into each set of the packet buffer
}
might be worth a try. I think you're loops are being overrun, not your main issue, but a confusing point for me nonetheless.
EDIT Everytime I look at it, I see stuff to be pulled out of loop.
I see a few problems.
You need to declare packet_buffer as packet_buffer[100][1025]. 1 byte for the ID and 1024 bytes for the buffer data. Total = 1025.
Your for (j...) loop is looping from 0 to 1024. It should go from 0 to 1023 so that you transfer a total of 1024 bytes in each iteration of the i loop.
You need to add 1024 to k at the end of the j loop, not 1025. Similarly, you can set j=1024 to exit the j loop when count==size.
You want to use &packet_buffer[temp][1] in your printf since the first char is an index and not from the file buffer.
Figured it out, since we were storing 0 in the beginning of the first packet as a char it stored a '\00' character which made the functions we were using to write the file stop as soon as it starts. Thanks for the all the help.
I have a binary file and I will be using fread to read the data from this binary file into an array of structures.
However, I don't know what value to pass to fread as its second argument. I know the file size is 536870912 bits. The binary file was constructed on the basis of being accessed for a 512^3 array. This means each data entry is of type float in the binary file with 4 bytes specified for each data element.
I made an error with the mention of bits. I read what was outputted by a C program finding the size of the file - it outputted 536870912 bits! Apologies to anyone confused.
Here is the code i'm using to read the data from the binary file into my arrary of structures (a simplified structure - there are 10 other parameters!)
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <math.h>
// Define the model structure
struct model {
float density;
};
// Entry point for the program
int main () {
int counter;
long lSize;
char * buffer;
size_t result;
FILE *pFile;
int i,j,k,ibox; /* Loop indices for the physical grid */
struct model ***mymodel;
pFile = fopen("core1_dens_0107.bin","rb");
if (pFile == NULL) { printf("Unable to open density file!"); exit(1); }
// obtain file size:
fseek (pFile , 0 , SEEK_END);
lSize = ftell (pFile);
rewind (pFile);
printf( "File size : %lu Bits \n", lSize );
for ( j = 0 ; j < 512 ; j++ ) {
for ( k = 0; k < 512; k++ ) {
for ( i = 0; i < 512; i++ ) {
fread(&mymodel[i][j][k].density,4,1,pFile);
printf("%f \n",mymodel[i][j][k].density);
}
}
}
fclose(pFile);
return 0;
}
Supposing you have already opened the file and you have your file descriptor myStream, it should be as simple as this:
#define MY_DIM = 512; ///Maybe you want to play safe and make it a little bit larger? Up to you
float buffer[MY_DIM][MY_DIM][MY_DIM];
size_t readBytes;
int i,j,k;
for (k = 0; k < MY_DIM; k++)
for (j = 0; j < MY_DIM; j++) {
readBytes = fread((void*) (buffer[k][j]), sizeof float, MY_DIM, myStream); //I am not sure the (void*) conversion is necessary
if (readBytes < MY_DIM) //I unexpectedly reached the end of the file,
goto endOfTheLoop; //without reading all the data I needed for int
//You could also print a warning message
}
endOfTheLoop:
//Now close the input file, use fclose or something
//Now that you have read all the data, you have to put it in your array of struct:
for (k = 0; k < MY_DIM; k++)
for (j = 0; j < MY_DIM; j++)
for (i = 0; i < MY_DIM; i++)
mymodel[k][j][i].density = buffer[k][j][i];
You can pass whatever value of the 2nd argument is most convenient for your program. If you want to process the file one structure at a time, do:
nread = fread(&your_struct, 1, sizeof yourstruct, stream);
If you have an array of structures, e.g.
struct foo your_struct[STRUCT_COUNT];
you can do:
nread = fread(your_struct, STRUCT_COUNT, sizeof *your_struct, stream);
size_t fread(void *ptr, size_t size, size_t nmemb, FILE * stream );
will attempt to read nmemb blocks of size bytes each. It will guarantee that no partial blocks are read. If your blocks are 4 bit long then I suggest you read them byte by byte, otherwise use the size argument to specify the block size.
For instance
fread(buffer, 1, 1024, stdin);
will attempt to read 1024 bytes, but may stop at any point.
fread(buffer, 4, 256, stdin);
will attempt to also read 1024 bytes, but in blocks of 4 bytes. 256 blocks total. It will guarantee that no partial blocks are read.
fread(buffer, 1024, 1, stdin);
will attempt to read one block of 1024 bytes. If it can not - nothing will be read.
If you wish to read in the entire file then you can do it in blocks of 4 via:
size_t read, read_now;
while (read < filesize && (read_now= fread(buffer +read, 4, (filesize - read) >> 2, in)) != EOF)
read += read_now;
of you can attempt to read the whole thing in one go:
fread(buffer, filesize, 1, in);
I have to read a png file(in binary) and i have to get out height and width of the picture(starting as 17 and long 8bytes together).
i tried
FILE* picture = fopen("test.png","rb");
fseek(picture, 16 , SEEK_SET);
int counter = 0;
while(counter < 8){
fscanf(picture, "%d", temp[counter]);
counter++;
}
but no luck
You should never read binary files using the file-library, but rather the system-calls read and write.
void* buf = malloc(1024);
memset(buf,0,1024);
int picturefd = open("test.png",O_RDONLY);
lseek(picturefd, 16 , SEEK_SET);
if(read(picturefd, buf, 8) < 8){
//failed to read or eof reached
}
See here for read and here for lseek.