I'm trying to convert an vector of RGB image data (derived from a .png image) to YUV420p format using libav.
In the libav sample code the following is used to create a dummy image:
/* prepare a dummy image */
static void fill_yuv_image(AVFrame *pict, int frame_index, int width, int height)
{
int x, y, i;
i = frame_index;
/* Y */
for(y=0;y<height;y++) {
for(x=0;x<width;x++) {
pict->data[0][y * pict->linesize[0] + x] = x + y + i * 3;
}
}
/* Cb and Cr */
for(y=0;y<height/2;y++) {
for(x=0;x<width/2;x++) {
pict->data[1][y * pict->linesize[1] + x] = 128 + y + i * 2;
pict->data[2][y * pict->linesize[2] + x] = 64 + x + i * 5;
}
}
}
I'm not clear about a few things here:
Firstly, do I need to rearrange the RGB data in the input vector so that it's suitable for encoding as YUV420p?
Secondly, I understand that there's a Y value for every pixel and that the Cb and Cr values are used for four (2x2) pixels. What I don't understand is how the RGB data gets "reduced" to the Cb and Cr values - is there an example of how to do this anywhere?
I'm not entirely sure what you're trying to achieve exactly, so I'll just directly answer your questions as best I can (feel free to follow up with clarifying comments):
1) You will be transforming the RGB data to YUV which will involve some rearrangement. The packed RGB data is fine where it is. You don't really need to adjust it. Actually, it would probably be better to leave it packed the way it is for cache locality reasons.
2) As you already understand, YUV 4:2:0 encodes a Y sample for each pixel but each 2x2 block shares a Cb and a Cr value. However, there is also YUV 4:4:4 data. This is where each pixel gets its own Y, Cb, and Cr sample. A simple strategy for converting RGB -> YUV 4:2:0 is to convert RGB -> YUV 4:4:4 and then average (arithmetic mean) each block of 2x2 Cb samples. There are other algorithms (like filters that involve more of the surrounding samples), but this should work if you're just experimenting with how this stuff works.
Another strategy for experimentation (and speed) is to only compute the Y plane and hold the Cb and Cr planes constant at 128. That will result in a grayscale image.
For real work, you would probably want to leverage the built-in conversion facilities that libav has to offer.
Related
I am attaching the picture example and my current code.
My question is: Can I make resizing/streching/interpolating single vertical bitmap strip faster
that using another for-loop.
The current Code looks very optimal:
for current strip size in the screen, iterate from start height to end height. Get corresponding
pixel from texture and add to output buffer. Add step to get another pixel.
here is an essential part of my code:
inline void RC_Raycast_Walls()
{
// casting ray for every width pixel
for (u_int16 rx = 0; rx < RC_render_width_i; ++rx)
{
// ..
// traversing thru map of grid
// finding intersecting point
// calculating height of strip in screen
// ..
// step size for nex pixel in texutr
float32 tex_step_y = RC_texture_size_f / (float32)pp_wall_height;
// starting texture coordinate
float32 tex_y = (float32)(pp_wall_start - RC_player_pitch - player_z_div_wall_distance - RC_render_height_d2_i + pp_wall_height_d2) * tex_step_y;
// drawing walls into buffer <- ENTERING ANOTHER LOOP only for SINGLE STRIP
for (int16 ry = pp_wall_start; ry < pp_wall_end; ++ry)
{
// cast the texture coordinate to integer, and mask with (texHeight - 1) in case of overflow
u_int16 tex_y_safe = (u_int16)tex_y & RC_texture_size_m1_i;
tex_y += tex_step_y;
u_int32 texture_current_pixel = texture_pixels[RC_texture_size_i * tex_y_safe + tex_x];
u_int32 output_pixel_index = rx + ry * RC_render_width_i;
output_buffer[output_pixel_index] =
(((texture_current_pixel >> 16 & 0x0ff) * intensity_value) >> 8) << 16 |
(((texture_current_pixel >> 8 & 0x0ff) * intensity_value) >> 8) << 8 |
(((texture_current_pixel & 0x0ff) * intensity_value) >> 8);
}
}
}
Maybe some bigger stepping like 2 instead of 1, got then every second line empty,
but adding another line of code that could fil that empty space results the same performance..
I would not like to have doubled pixels and interpolating between two of them I think would take even
longer. ??
Thank You in Advance!
ps.
Its based on Lodev Raycaster algorithm:
https://lodev.org/cgtutor/raycasting.html
You do not need floats at all
You can use DDA on integers without multiplication and division. These days floating is not that slow as it used to but your conversion between float and int might be ... See these QAs (both use this kind of DDA:
DDA line with subpixel
DDA based rendering routines
use LUT for applying Intensity
Looks like each color channel c is 8 bit and intensity i is fixed point in range <0,1> so you can precompute every combination into something like this:
u_int8 LUT[256][256]
for (int c=0;c<256;c++)
for (int i=0;i<256;i++)
LUT[c][i]=((c*i)>>8)
use pointers or union to access RGB channels instead of bit operations
My favorite is union:
union color
{
u_int32 dd; // 1x 32bit RGBA
u_int16 dw[2]; // 2x 16bit
u_int8 db[4]; // 4x 8bit (individual channels)
};
texture coordinates
Again looks like you are doing too many operations. for example [RC_texture_size_i * tex_y_safe + tex_x] if your texture size is 128 you can bitshift lef by 7 bits instead of multiplication. Yes on modern CPUs is this not an issue however the whole thing can be replaced by simple LUT. You can remember pointer to each horizontal ScanLine of texture and rewrite to [tex_y_safe][tex_x]
So based on #2,#3 rewrite your color computation to this:
color c;
c.dd=texture_current_pixel;
c.db[0]=LUT[c.db[0]][intensity_value];
c.db[1]=LUT[c.db[1]][intensity_value];
c.db[2]=LUT[c.db[2]][intensity_value];
output_buffer[output_pixel_index]=c.dd;
As you can see its just bunch of memory transfers instead of multiple bit-shifts,bit-masks and bit-or operations. You can also use pointer of color instead of texture_current_pixel and output_buffer[output_pixel_index] to speed up little more.
And finally see this:
Ray Casting with different height size
Which is my version of the raycast using VCL.
Now before changing anything measure the performance you got now by measuring the time it needs to render. Then after each change in the code measure if it actually improve performance or not. In case it didn't use old version of code as predicting what is fast on nowadays platforms is sometimes hard.
Also for resize much better visual results are obtained by using mipmaps ... that usually eliminates the weird noise while moving
I've been reverse engineering a program and recently came across a function that is intended to create a sort of translucent-looking color to be used for text selections. It does this by converting RGB to YUV, alters the Y (luma?) component, then converts back to RGB.
uint32_t CalcSelectionColor(uint32_t bgr)
{
double r,g,b;
double y,u,v;
r = (bgr >> 0) & 0xFF;
g = (bgr >> 8) & 0xFF;
b = (bgr >> 16) & 0xFF;
/* RGB to YUV */
y = 0.299*r + 0.587*g + 0.114*b;
u = (b-y) * 0.565 * 0.5;
v = (r-y) * 0.713 * 0.5;
/* lower brightness? */
y = 255.0 - y;
/* YUV to RGB */
r = y + 1.403*v;
g = y - 0.344*u - 0.714*v;
b = y + 1.77*u;
return ((uint8_t)(b) << 16) | ((uint8_t)(g) << 8) | ((uint8_t)(r));
}
As someone with very limited knowledge of computer graphics, I'd just like a bit more detail of what it does between the conversions, and the actually intended effect in a broader sense. Is this a common approach of adjusting brightness of a color or something? If I pass in 0x00FF00, the result I get is 0x1E9D1E
The formulas used in this code are similar to Julien transformation from RGB to YUV and back:
Transformation from RGB to YUV:
Y = 0.299R + 0.587G + 0.114B
U'= (B-Y)*0.565
V'= (R-Y)*0.713
Transformation from YUV to RGB:
R = Y + 1.403V'
G = Y - 0.344U' - 0.714V'
B = Y + 1.770U'
However, the formulas in your code are a bit different. While the back transformation is the same, the forward transform has an additional multiplier 0.5 for both U and V components. There is also a trivial manipulation with the brightness component
y = 255.0 - y
which simply inverses the brightness. So, what happens here?
If you use normal Julien RGB->YUV transform, you get a representation for your color as a combination of brightness Y and two color tone components U and V, which define the color as shown on this picture:
However, in your code you also multiply both U and V components by 0.5. This means, that on this UV plane you move from any given color two times closer to the point of origin (0, 0). For example, if the initial color was A with UV coordinates (-0.4, 0.3), then you'll get a new color B with UV coordinates (-0.2, 0.15). Similarly, the color C (0.2, -0.3) becomes color D (0.1, -0.15):
After that you inverse the brightness of the color, making dark colors bright and bright colors dark. This is the effect of your code.
It's not terribly common, but it's a very good approach. Commonly used models like HSL/HSV don't represent intensity correctly and have some weird piecewise-linear stuff with hue/color going on. YUV is a really good colorspace, representing intensity along one axis and chroma (hue/color) in a perpendicular plane.
Normally modifying Y without also adjusting (at least clamping) U and V is somewhat dubious, because near the extremes (Y=0 black, Y=full white) U and V have limited range (no range at all at the endpoints). Otherwise applying them will take you outside of the RGB cube and result in bogus clipped results when you go back to RGB. But here the trick is very clever. The code is inverting Y while keeping chroma fixed, so the incoming range limits on U and V near black will automatically ensure they're roughly correct in the output, and vice versa.
As Alex noted, the code here is also halving the chroma values, reducing color saturation. This was probably to avoid the above mentioned clipping issue, but it's not needed. But maybe it's part of the intended visual effect too.
So, TL;DR: the effect is inverting intensity/luma and halving saturation.
In OpenGL ES 2.0 after reading the framebuffer and converting it to RGBA frames. I want to convert it to YUV format.
I tried using this table. Which ignore alpha component. When i do that and see the YUV frame generated its distorted.
Can anyone help me please
yuvdata[i * j *1]= (0.257)*memory[i*j*1] + (0.504)*memory[i*j*2]+(0.098)*memory[i*j*3]+16;
yuvdata[i * j *3]= (0.439)*memory[i*j*1] - (0.368)*memory[i*j*2] -(0.071)*memory[i*j*3]+128;
yuvdata[i * j *2]= -(0.148)*memory[i*j*1] - (0.291)*memory[i*j*2] +(0.439)*memory[i*j*3]+128;
`
Normalization did not help
memory store rgba
and yuv is free space to store yuv
i used rgb to yuv conversion ignoring alpha component
Friends this problem is resolved. This article is awesomeyuv2rgb. Thanks to Viktor Latypov and Mārtiņš Možeiko
"YUV" is not a complete format.
From this wikipedia article you can get the conversions of YUV411,YUV422,YUV420p to YUV444. Combine the inverse of these transforms with your RGB conversion and you'll get the result.
The thing you are missing: one RGB triple may produce a number (not one) of YUV components this way.
YUV444 3 bytes per pixel
YUV422 4 bytes per 2 pixels
YUV411 6 bytes per 4 pixels
YUV420p 6 bytes per 4 pixels, reordered
First, YUV422
Y'UV422 to RGB888 conversion
Input: Read 4 bytes of Y'UV (u, y1, v, y2 )
Output: Writes 6 bytes of RGB (R, G, B, R, G, B)
Then YUV4111
Y'UV411 to RGB888 conversion
Input: Read 6 bytes of Y'UV
Output: Writes 12 bytes of RGB
// Extract YUV components
u = yuv[0];
y1 = yuv[1];
y2 = yuv[2];
v = yuv[3];
y3 = yuv[4];
y4 = yuv[5];
rgb1 = Y'UV444toRGB888(y1, u, v);
rgb2 = Y'UV444toRGB888(y2, u, v);
rgb3 = Y'UV444toRGB888(y3, u, v);
rgb4 = Y'UV444toRGB888(y4, u, v);
Similar with 420p, but the YUV values are distributed over the rectangle there - see the Wikipedia's diagram and image for that.
Basically, you should fetch 4 RGB pixels, convert each one of them to YUV (using your hopefully valid 444 converter) and then store the YUV[4] array in a tricky way shown at the wikipedia.
Are you sure you are accessing memory array correctly right? Shouldn't it be something like this:
R=memory[(i*width + j)*4+0]
G=memory[(i*width + j)*4+1]
B=memory[(i*width + j)*4+3]
or swap 0 with 3, if you have BGRA.
This is only marginally programming related - has much more to do w/ colors and their representation.
I am working on a very low level app. I have an array of bytes in memory. Those are characters. They were rendered with anti-aliasing: they have values from 0 to 255, 0 being fully transparent and 255 totally opaque (alpha, if you wish).
I am having trouble conceiving an algorithm for the rendering of this font. I'm doing the following for each pixel:
// intensity is the weight I talked about: 0 to 255
intensity = glyphs[text[i]][x + GLYPH_WIDTH*y];
if (intensity == 255)
continue; // Don't draw it, fully transparent
else if (intensity == 0)
setPixel(x + xi, y + yi, color, base); // Fully opaque, can draw original color
else { // Here's the tricky part
// Get the pixel in the destination for averaging purposes
pixel = getPixel(x + xi, y + yi, base);
// transfer is an int for calculations
transfer = (int) ((float)((float) (255.0 - (float) intensity/255.0) * (float) color.red + (float) pixel.red)/2); // This is my attempt at averaging
newPixel.red = (Byte) transfer;
transfer = (int) ((float)((float) (255.0 - (float) intensity/255.0) * (float) color.green + (float) pixel.green)/2);
newPixel.green = (Byte) transfer;
// transfer = (int) ((float) ((float) 255.0 - (float) intensity)/255.0 * (((float) color.blue) + (float) pixel.blue)/2);
transfer = (int) ((float)((float) (255.0 - (float) intensity/255.0) * (float) color.blue + (float) pixel.blue)/2);
newPixel.blue = (Byte) transfer;
// Set the newpixel in the desired mem. position
setPixel(x+xi, y+yi, newPixel, base);
}
The results, as you can see, are less than desirable. That is a very zoomed in image, at 1:1 scale it looks like the text has a green "aura".
Any idea for how to properly compute this would be greatly appreciated.
Thanks for your time!
You need to blend the background and foreground colours. A-la:
pixelColour = newColour * intensity + backgroundColour * (1 - intensity)
By the way, this is a really slow way of rendering and blending fonts. You should instead render all the characters of the font to an off-screen surface with all the properties you need, and then use that as a texture to render to other surfaces when you need text.
Edit:
This doesn't look right:
(255.0 - (float) intensity/255.0)
It should instead be:
(255.0 - (float) intensity)/255.0
I believe that "aura" is caused by anti aliasing. The technique averages pixels with their neighbors.
I realize you don't seem to be using OpenGL but this chapter might help explain some of the theory. Wish I had a better answer, but hopefully this points you in the right direction. My first attempt would be to disable Antialiasing since it seems to do more harm than good. There is probably a better solution than that though.
It may be too much complicated to doing alpha blending pixel by pixel because current pixel value modifies next pixel value.
I would redesign the algorithm with the thinking of box wise blending.
With many getPixel calling for a single glyph, you can't produce proper target image.
I want to read the RGB values for each pixel from a raw image. Can someone tell me how to achieve this? Thanks for help!
the format of my raw image is .CR2 which come from camera.
Assuming the image is w * h pixels, and stored in true "packed" RGB format with no alpha component, each pixel will require three bytes.
In memory, the first line of the image might be represented in awesome ASCII graphics like this:
R0 G0 B0 R1 G1 B1 R2 G2 B2 ... R(w-1) G(w-1) B(w-1)
Here, each Rn Gn and Bn represents a single byte, giving the red, green or blue component of pixel n of that scanline. Note that the order of the bytes might be different for different "raw" formats; there's no agreed-upon world standard. Different environments (graphics cards, cameras, ...) do it differently for whatever reason, you simply have to know the layout.
Reading out a pixel can then be done by this function:
typedef unsigned char byte;
void get_pixel(const byte *image, unsigned int w,
unsigned int x,
unsigned int y,
byte *red, byte *green, byte *blue)
{
/* Compute pointer to first (red) byte of the desired pixel. */
const byte * pixel = image + w * y * 3 + 3 * x;
/* Copy R, G and B to outputs. */
*red = pixel[0];
*green = pixel[1];
*blue = pixel[2];
}
Notice how the height of the image is not needed for this to work, and how the function is free from bounds-checking. A production-quality function might be more armor-plated.
Update If you're worried this approach will be too slow, you can of course just loop over the pixels, instead:
unsigned int x, y;
const byte *pixel = /* ... assumed to be pointing at the data as per above */
for(y = 0; y < h; ++y)
{
for(x = 0; x < w; ++x, pixel += 3)
{
const byte red = pixel[0], green = pixel[1], blue = pixel[2];
/* Do something with the current pixel. */
}
}
None of the methods posted so far are likely to work with a camera "raw" file. The file formats for raw files are proprietary to each manufacturer, and may contain exposure data, calibration constants, and white balance information, in addition to the pixel data, which will likely be in a packed format where each pixel can takes up more than one byte, but less than two.
I'm sure there are open-source raw file converter programs out there that you could consult to find out the algorithms to use, but I don't know of any off the top of my head.
Just thought of an additional complication. The raw file does not store RGB values for each pixel. Each pixel records only one color. The other two colors have to be interpolated from heighboring pixels. You'll definitely be better off finding a program or library that works with your camera.
A RAW image is an uncompressed format, so you just have to point where your pixel is (skipping any possible header, and then adding the size of the pixel times the number columns times the number of row plus the number of the colum), and then read whatever binary data is giving a meaningful format to the layout of the data (with masks and shifts, you know).
That's the general procedure, for your current format you'll have to check the details.