I use jpeg library v8d from Independent JPEG Group and I want to change the way jpeg decompression reads and processes data.
In the djpeg main(), only one scanline/row at a time is read and processed in each jpeg_read_scanlines() call. So, to read entire image this functions is called until all lines are read and processed:
while (cinfo.output_scanline < cinfo.output_height) {
num_scanlines = jpeg_read_scanlines(&cinfo, dest_mgr->buffer,
dest_mgr->buffer_height); //read and process
(*dest_mgr->put_pixel_rows) (&cinfo, dest_mgr, num_scanlines); //write to file
}
But I would like to read the entire image once and store it in the memory and then process the entire image from memory. By reading libjpeg.txt, I found out this is possible: "You can process an entire image in one call if you have it all in memory, but usually it's simplest to process one scanline at a time."
Even though I made some progress, I couldn't make it completely work. I can now read a couple of rows once by increasing pub.buffer_height value and pub.buffer size, but no matter how large pub.buffer_height and pub.buffer are, only a couple of lines are read in each jpeg_read_scanlines() call. Any thoughts on this?
only a couple of lines are read in each jpeg_read_scanlines()
Yes, so you call it in a loop. Here's a loop that grabs one scanline at a time:
unsigned char *rowp[1], *pixdata = ...;
unsigned rowbytes = ..., height = ...;
while (cinfo.output_scanline < height) {
rowp[0] = pixdata + cinfo.output_scanline * rowbytes;
jpeg_read_scanlines(&cinfo, rowp, 1);
}
Once the loop exits, you have the entire image.
Related
I'm porting some C code that loads sprites from files containing multiple bitmaps. Basically the code fopens the file, fgetcs some header info, then freads the bitmap data. I can see that the fgetcs are returning proper data, but the outcome of the fread is null. Here's the code - fname does exist, the path is correct, fil is non-zero, num is the number of sprites in the file (encoded into the header, little-endian), pak is an array of sprites, sprite is a typedef of width, height and bits, and new_sprite inits one for you.
FILE *fil;
uint8 *buffu;
uint8 read;
int32 x,num;
int32 w,h,c;
fil = fopen(fname, "rb");
if (!fil) return NULL;
num = fgetc(fil);
num += fgetc(fil)*256;
if (num > max) max = num;
for (x=0;x<max;x++) {
// header
w=fgetc(fil);
w+=fgetc(fil)*256;
h=fgetc(fil);
h+=fgetc(fil)*256;
fgetc(fil); // stuff we don't use
fgetc(fil);
fgetc(fil);
fgetc(fil);
// body
buffu = (uint8*)malloc(w * h);
read=fread(buffu,1,w*h,fil);
pak->spr[x]=new_sprite(w,h);
memcpy(pak->spr[x]->data, buffu, w*h);
// done
free(buffu);
}
I've stepped through this code line by line, and I can see that w and h are getting set up properly, and read=4096, which is the right number of bits. However, buffer is "" after the fread, so of course memcpy does nothing useful and my pak is filled with empty sprites.
My apologies for what is surely a totally noob question, but I normally use Cocoa so this pure-C file handling is new to me. I looked all over for examples of fread, and they all look like the one here - which apparently works fine on Win32.
Since fgetc seems to work, you could try this as a test
int each;
int byte;
//body
buffu = malloc(w * h);
for (each = 0; each < w*h; each++) {
byte = fgetc(fil);
if ( byte == EOF) {
printf("End of file\n");
break;
}
buffu[each] = (uint8)byte;
printf ("byte: %d each: %d\n", byte, each);
}
pak->spr[x]=new_sprite(w,h);
memcpy(pak->spr[x]->data, buffu, w*h);
// done
You say:
However, buffer is "" after the fread, so of course memcpy does nothing useful
But that is not true at all. memcpy() is not a string function, it will copy the requested number of bytes. Every time. If that isn't "useful", then something else is wrong.
Your buffer, when treated as a string (which it is not, it's a bunch of binary data) will look like an empty string if the first byte happens to be 0. The remaining 4095 bytes can be whatever, to C's string printing functions it will look "empty".
I am new to OpenGL and I am currently reading OpenGL Superbible. In the following code, it creates 3 buffer names, each one is bound to GL_TEXTURE_BUFFER_ARB, then some data is buffered.
My question is, when a new buffer is bound to GL_TEXTURE_BUFFER_ARB, what happens to the previous bufferde data? How is it used?
// Create 3 new buffer objects
glGenBuffers(3,texBO);
glGenTextures(1, &texBOTexture);
int count = 0;
float* fileData = 0;
// Load first texBO with a tangent-like curve, 1024 values
fileData = LoadFloatData(“LumTan.data”, &count);
if (count > 0)
{
glBindBuffer(GL_TEXTURE_BUFFER_ARB, texBO[0]);
glBufferData(GL_TEXTURE_BUFFER_ARB, sizeof(float)*count,
fileData, GL_STATIC_DRAW);
delete fileData;
}
// Load second texBO with a sine-like curve, 1024 values
fileData = LoadFloatData(“LumSin.data”, &count);
if (count > 0)
{
glBindBuffer(GL_TEXTURE_BUFFER_ARB, texBO[1]);
glBufferData(GL_TEXTURE_BUFFER_ARB, sizeof(float)*count,
fileData, GL_STATIC_DRAW);
delete fileData;
}
// Load third texBO with a linear curve, 1024 values
fileData = LoadFloatData(“LumLinear.data”, &count);
if (count > 0)
{
glBindBuffer(GL_TEXTURE_BUFFER_ARB, texBO[2]);
glBufferData(GL_TEXTURE_BUFFER_ARB, sizeof(float)*count,
fileData, GL_STATIC_DRAW);
delete fileData;
}
// Load the Tan ramp first
glBindBuffer(GL_TEXTURE_BUFFER_ARB, 0);
glActiveTexture(GL_TEXTURE1);
glBindTexture(GL_TEXTURE_BUFFER_ARB, texBOTexture);
glTexBuffer(GL_TEXTURE_BUFFER_ARB, GL_R32F, texBO[0]);
glActiveTexture(GL_TEXTURE0);
Actually I don't understand where the data in these buffers is actually used. To me, the code looks like:
int x = 3;
x = 7; //what about 3?
x = 5; //what about 7?
Moreover, after doing this (binding buffers, then buffering data into these buffers) to the three buffers, all the buffers are unbound from GL_TEXTURE_BUFFER_ARB by binding 0 to it.
All OpenGL object are represented by an unique id.
So if you create a Shader, Programm, Buffer, ... then you get an ID for these objects. These Objects live on the graphic card until you call delete them with glDelete[...] the commands you call for those objects are applied to the currently bound object glBind[...].
You can imagine it like a dialog between you and the gpu:
You tell the GPU with which object you currently want to work
The GPU activate this object for you
You then tell the GPU what you want to do with the the currently active object
The GPU applies this information to the active object.
With glGenBuffers(3,texBO) you create 3 Buffers are create and their ids are stored in texBO.
glBindBuffer(GL_TEXTURE_BUFFER_ARB, texBO[0]);:
Active the buffer with that is represented by the id stored in texBO[0]
glBufferData(GL_TEXTURE_BUFFER_ARB, sizeof(float)*count,
fileData, GL_STATIC_DRAW);:
Popolate the currently active Buffer (in this case texBO[0]) with the data stored in fileData.
glBindBuffer(GL_TEXTURE_BUFFER_ARB, texBO[1]);:
Now activates the buffer with that is represented by the id stored in texBO[1] the other buffer still exists but is not currently bound.
I have a problem when I write a large amount of data <2GB to a file. First ~1.4GB data are written fast (100 MB/s) than the code become really slow (0-2 MB/s).
My code (simplified) is:
//FileOptions FILE_FLAG_NO_BUFFERING = (FileOptions)0x20000000;
FileOptions fileOptions = FileOptions.SequentialScan;
int fileBufferSize = 1024 * 1024;
byte[] Buffer = new byte[32768];
Random random = new Random();
long fileSize = 2588490188;
long totalByteWritten = 0;
using (FileStream fs = File.Create(#"c:\test\test.bin", fileBufferSize, fileOptions))
{
while (totalByteWritten < fileSize)
{
random.NextBytes(Buffer);
fs.Write(Buffer, 0, Buffer.Length);
totalByteWritten += Buffer.Length;
//Thread.Sleep(10);
}
}
I think there is an issue related to caching problem, in fact during "fast write performance" RAM used increase as well, when RAM usage stop to increase there is a drop in performance.
What I have tried:
change to async write
->no significantly change
change array buffer size
->no significantly change
change fileBufferSize
->no significantly change, but with a large buffer ~100MB, write performance is fast and when RAM usage stop to increase, write performance goes to 0 and than, after a while, goes back to 100MB, it seams that cache buffer is "flushed"
change fileOption to WriteThrough
->performance are always slow..
adding after xx loops fs.Flush(true)
->no significantly change
uncomment Thread.Sleep(10)
->write speed is always good.....this is strange
Is it somehow trying to write before it's finished writing the previous chunk and getting in a mess? (seems unlikely, but it's very odd that the Thread.Sleep should speed it up and this might explain it). What happens if you modify the code inside the using statement to lock the filestream, like this?
using (FileStream fs = File.Create(#"c:\testing\test.bin", fileBufferSize, fileOptions))
{
while (fs.Position < fileBufferSize)
{
lock(fs) // this is the bit I have added to try to speed it up
{
random.NextBytes(Buffer);
fs.Write(Buffer, 0, Buffer.Length);
}
}
}
EDIT: I have tweaked your example code to include the while loop required to make it write a file of the correct size.
Incidentally, when I run the sample code it is very quick with or without the lock statement and adding the sleep slows it down significantly.
I want to delete a string from a particular position in the file. Is thre a function to do that?
Can I delete last line of a file through a function?
You have two option
To read whole file, remove what you need and write it back
If the file is big, read file sequentially, remove given part, and move content after that forward
No there is no such function that will let you do this directly on a file.
You should load up the file content in memory and modify there and write back to file.
I don't feel like looking up all the io functions, so here's pseudo-c on how to implement option 2 of ArsenMkrt's answer
char buffer[N]; // N >= 1
int str_start_pos = starting position of the string to remove
int str_end_pos = ending position of the string to remove
int file_size = the size of the file in bytes
int copy_to = str_start_pos
int copy_from = str_end_pos + 1
while(copy_from < file_size){
set_file_pos(file, copy_from)
int bytes_read = read(buffer, N, file)
copy_from += bytes_read
set_file_pos(file, copy_to)
write(buffer, file, bytes_read)
copy_to += bytes_read
}
truncate_file(file,file_size - (str_end_pos - str_start_pos + 1))
something to that effect
Is there any way to figure out the length of a .dat file (in terms of rows) without loading the file into the workspace?
Row Counter -- only loads one character per row:
Nrows = numel(textread('mydata.txt','%1c%*[^\n]'))
or file length (Matlab):
datfileh = fopen(fullfile(path, filename));
fseek(datfileh, 0,'eof');
filelength = ftell(datfileh);
fclose(datfileh);
I'm assuming you are working with text files, since you mentioned finding the number of rows.
Here's one solution:
fid = fopen('your_file.dat','rt');
nLines = 0;
while (fgets(fid) ~= -1),
nLines = nLines+1;
end
fclose(fid);
This uses FGETS to read each line, counting the number of lines it reads. Note that the data from the file is never saved to the workspace, it is simply used in the conditional check for the while loop.
It's also worth bearing in mind that you can use your file system's in-built commands, so on linux you could use the command
[s,w] = system('wc -l your_file.dat');
and then get the number of lines from the returned text (which is stored in w). (I don't think there's an equivalent command under Windows.)