I'm trying to build an image viewer for 16-bit PNG images with WPF. My idea was to load the images with PngBitmapDecoder, then put them into an Image control, and control the brightness/contrast with a pixel shader.
However, I noticed that the input to the pixel shader seems to be converted to 8 bit already. Is that a known limitation of WPF or did I make a mistake somewhere? (I checked that with a black/white gradient image that I created in Photoshop and which is verified as 16-bit image)
Here's the code to load the image (to make sure that I load with the full 16-bit range, just writing Source="test.png" in the Image control loads it as 8-bit)
BitmapSource bitmap;
using (Stream s = File.OpenRead("test.png"))
{
PngBitmapDecoder decoder = new PngBitmapDecoder(s,BitmapCreateOptions.PreservePixelFormat, BitmapCacheOption.OnLoad);
bitmap = decoder.Frames[0];
}
if (bitmap.Format != PixelFormats.Rgba64)
MessageBox.Show("Pixel format " + bitmap.Format + " is not supported. ");
bitmap.Freeze();
image.Source = bitmap;
I created the pixel shader with the great Shazzam shader effect tool.
sampler2D implicitInput : register(s0);
float MinValue : register(c0);
float MaxValue : register(c1);
float4 main(float2 uv : TEXCOORD) : COLOR
{
float4 color = tex2D(implicitInput, uv);
float t = 1.0f / (MaxValue-MinValue);
float4 result;
result.r = (color.r - MinValue) * t;
result.g = (color.g - MinValue) * t;
result.b = (color.b - MinValue) * t;
result.a = color.a;
return result;
}
And integrated the shader into XAML like this:
<Image Name="image" Stretch="Uniform">
<Image.Effect>
<shaders:AutoGenShaderEffect x:Name="MinMaxShader" Minvalue="0.0" Maxvalue="1.0>
</shaders:AutoGenShaderEffect>
</Image.Effect>
</Image>
I just got a response from Microsoft. It's really a limitation in the WPF rendering system. I'll try D3DImage as a workaround.
Related
Is it possible to have DropShadowEffect to ignore certain colors when rendering shadow? To have sort of masked (color selective) shadow?
My problem is what shadow can be assigned to whole visual element (graph). It looks like this:
And I want
Notice grid lines without shadow (except 0,0 ones). This can be achieved by having 2 synchronized in zooming/offset graphs, one without shadow effect containing grid and another with shadow containing the rest. But I am not very happy about this solution (I predict lots of problems in the future with that solution). So I'd rather prefer to modify DropShadowEffect somehow.
I can create and use ShaderEffect, but I have no knowledge of how to program shaders to have actual shadow effect (if it can be produced by shaders at all).
Perhaps there is much easier way of doing something with DropShadowEffect itself? Anyone?
Edit
I tried to make shader effect:
sampler2D _input : register(s0);
float _width : register(C0);
float _height : register(C1);
float _depth : register(C2); // shadow depth
float4 main(float2 uv : TEXCOORD) : COLOR
{
// get pixel size
float2 pixel = {1 / _width, 1 / _height};
// find color at offset
float2 offset = float2(uv.x - pixel.x * _depth, uv.y - pixel.y * _depth);
float4 color = tex2D(_input, offset);
// convert to gray?
//float gray = dot(color, float4(0.1, 0.1, 0.1, 0));
//color = float4(gray, gray, gray, 1);
// saturate?
//color = saturate(color);
return tex2D(_input, uv) + color;
}
But fail at everything.
Edit
Here is screenshot of graph appearance, which I like (to those, who try to convince me not to do this):
Currently it is achieved by having special Graph which has template
<Border x:Name="PART_Border" BorderThickness="1" BorderBrush="Gray" CornerRadius="4" Background="White">
<Grid>
<Image x:Name="PART_ImageBack" Stretch="None"/>
<Image x:Name="PART_ImageFront" Stretch="None">
<Image.Effect>
<DropShadowEffect Opacity="0.3"/>
</Image.Effect>
</Image>
</Grid>
</Border>
Everything is rendered onto PART_ImageFront (with shadow), while grid is rendered onto PART_ImageBack (without shadow). Performance-wise it is still good.
I have zero experience with pixel shaders, but here's my quick and dirty attempt at a shadow effect that ignores "uncolored" pixels:
sampler2D _input : register(s0);
float _width : register(C0);
float _height : register(C1);
float _depth : register(C2);
float _opacity : register(C3);
float3 rgb_to_hsv(float3 RGB) {
float r = RGB.x;
float g = RGB.y;
float b = RGB.z;
float minChannel = min(r, min(g, b));
float maxChannel = max(r, max(g, b));
float h = 0;
float s = 0;
float v = maxChannel;
float delta = maxChannel - minChannel;
if (delta != 0) {
s = delta / v;
if (r == v) h = (g - b) / delta;
else if (g == v) h = 2 + (b - r) / delta;
else if (b == v) h = 4 + (r - g) / delta;
}
return float3(h, s, v);
}
float4 main(float2 uv : TEXCOORD) : COLOR {
float width = _width; // 512;
float height = _height; // 512;
float depth = _depth; // 3;
float opacity = _opacity; // 0.25;
float2 pixel = { 1 / width, 1 / height };
float2 offset = float2(uv.x - pixel.x * depth, uv.y - pixel.y * depth);
float4 srcColor = tex2D(_input, offset);
float3 srcHsv = rgb_to_hsv(srcColor);
float4 dstColor = tex2D(_input, uv);
// add shadow for colored pixels only
// tweak saturation threshold as necessary
if (srcHsv.y >= 0.1) {
float gray = dot(srcColor, float4(0.1, 0.1, 0.1, 0.0));
float4 multiplier = float4(gray, gray, gray, opacity * srcColor.a);
return dstColor + (float4(0.1, 0.1, 0.1, 1.0) * multiplier);
}
return dstColor;
}
Here it is in action against a (totally legit) chart that I drew in Blend with the pencil tool:
The shader effect is applied on the root panel containing the axes, grid lines, and series lines, and it generates a shadow only for the series lines.
I don't think it's realistic to expect a shader to be able to apply a shadow to the axes and labels while ignoring the grid lines; the antialiasing on the text is bound to intersect the color/saturation range of the grid lines. I think applying the shadow to just the series lines is cleaner and more aesthetically pleasing anyway.
sampler2D input : register(s0);
float4 main(float2 uv : TEXCOORD) : COLOR {
float4 Color;
Color = tex2D( input , uv.xy);
return Color;
}
This is a basic 'do nothing' shader. the line with the text2D call takes the color that normally would be plotted at the current location. (And in this case simply returns it)
Instead of sampling uv.xy you could add an offset vector to uv.xy and return that color. This would shift the entire image in the direction of the offset vector.
You could combine these two:
sample the uv.xy, if set to a visible color plot that color (this will keep all the lines visible at the right location)
if it is transparent, sample a bit to the top left. if it is set to a color you want to have a shadow: return the shadow color.
Step 2. can be changed into: if it is set to a color you do not want to have a shadow, return a transparent color.
The offset and the colors to test and to use as shadow color could be parameters of the effect.
I strongly suggest to play around with Shazzam it will allow you to test your shader and it will generate the C# code for you.
Note that the uv coordinates are are not in pixels but scaled to 0.0 to 1.0.
Addition
A poor man's blur (anti-aliasing) could be obtained by sampling more pixels around the offset and calculating an average of the colors found that should cause a shadow. this will cause more pixels to receive a shadow.
To calculate the shadow color you could simply darken the existing color by multiplying it with a factor in between 0.0 (black) and 1.0 (original color)
By using the average from the blur you can multiply the shadow color again causing the blur to mix with the original color.
More precise (and expensive) would be to translate the rgb values to hls values and use 'official' darkening formulas to determine the shadow color.
I am trying to capture images from PDF and display them in my iPad application in a UIScrollView. App render the images to fullscreen (full size) when pdf page size is 1024 x 768 (standard 4:3 aspect ratio). But when the pdf page is of size 720 x 540 (same 4:3 aspect ratio but with less dimensions) I am seeing white border around the image. Below is the code snippet that i tried to capture the images.
CGPDFPageRef thePDFPageRef = CGPDFDocumentGetPage(pdfDocRef, pdfIterator);
CGColorSpaceRef rgb = CGColorSpaceCreateDeviceRGB(); // RGB color space
CGBitmapInfo bmi = (kCGBitmapByteOrder32Little | kCGImageAlphaPremultipliedLast);
thumbnailWidth = 1024;
thumbnailHeight = 768;
CGContextRef context = CGBitmapContextCreate(NULL, thumbnailWidth, thumbnailHeight, 8, 0, rgb, bmi);
if (context != NULL) // Must have a valid custom CGBitmap context to draw into
{
CGRect thumbRect = CGRectMake(0.0f, 0.0f, thumbnailWidth, thumbnailHeight); // Target thumb rect
CGContextSetRGBFillColor(context, 1.0f, 1.0f, 1.0f, 1.0f);
CGContextFillRect(context, thumbRect); // White fill
CGContextConcatCTM(context, CGPDFPageGetDrawingTransform(thePDFPageRef, kCGPDFCropBox, thumbRect, 0, true)); // Fit rect
CGContextSetRenderingIntent(context, kCGRenderingIntentDefault);
CGContextSetInterpolationQuality(context, kCGInterpolationHigh);
CGContextDrawPDFPage(context, thePDFPageRef); // Render the PDF page into the custom CGBitmap context
imageRef = CGBitmapContextCreateImage(context); // Create CGImage from custom CGBitmap context
CGContextRelease(context); // Release custom CGBitmap context reference
}
CGColorSpaceRelease(rgb); // Release device RGB color space reference
CGPDFPageRelease(thePDFPageRef);
}
UIImage *image = [UIImage imageWithCGImage:imageRef scale:scale orientation:0];
How can I auto scale to full screen even when pdf page size is smaller?
I found the solution as below with scaling to rectangle.
(1) Get PDF page rectangle using
CGRect pageRect = CGPDFPageGetBoxRect(page, kCGPDFMediaBox);
(2) Do scale the rectangle and apply transform to Identity Matrix
CGAffineTransform trans = CGAffineTransformIdentity;
trans = CGAffineTransformTranslate(trans, 0, pageRect.size.height);
trans = CGAffineTransformScale(trans, 1.0, -1.0);
rect = CGRectApplyAffineTransform(rect, trans);
(3) Use kCGPDFMediaBox instead of kCGPDFCropBox to avoid cropping of pdfs when size is different than standard.
I'm loading a BitmapImage from a BitmapSource and i always find the BitmapImage's format is Bgr32 instead of Bgra32 which the BitmapSource is. There is a similar thread on this here:
BitmapImage from file PixelFormat is always bgr32
but using BitmapCreateOptions.PreservePixelFormat didn't work for me as was suggested in the thread. here's what i'm doing:
// I have verified that b is a valid BitmapSource and its format is Bgra32
// the following code produces a file (testbmp2.bmp) with an alpha channel as expected
// placing a breakpoint here and querying b.Format in the Immediate window also produces
// a format of Bgra32
BmpBitmapEncoder test = new BmpBitmapEncoder();
FileStream stest = new FileStream(#"c:\temp\testbmp2.bmp", FileMode.Create);
test.Frames.Add(BitmapFrame.Create(b));
test.Save(stest);
stest.Close();
// however, this following snippet results in bmp.Format being Bgr32
BitmapImage bmp = new BitmapImage();
BmpBitmapEncoder encoder = new BmpBitmapEncoder();
MemoryStream stream = new MemoryStream();
encoder.Frames.Add(BitmapFrame.Create(b));
encoder.Save(stream);
bmp.BeginInit();
bmp.StreamSource = new MemoryStream(stream.ToArray());
bmp.CreateOptions = BitmapCreateOptions.PreservePixelFormat;
bmp.CacheOption = BitmapCacheOption.None;
bmp.EndInit();
stream.Close();
Is there a way to create a BitmapImage from the BitmapSource and preserver the alpha channel?
Update:
I used the same code posted in the the other thread to load testbmp2.bmp from file and after loading, the Format property is Bgr32.
BitmapImage b1 = new BitmapImage();
b1.BeginInit();
b1.UriSource = new Uri(#"c:\temp\testbmp2.bmp");
b1.CreateOptions = BitmapCreateOptions.PreservePixelFormat | BitmapCreateOptions.IgnoreImageCache;
b1.CacheOption = BitmapCacheOption.OnLoad;
b1.EndInit();
so maybe i'm not saving the FileStream/MemoryStream correctly? That doesn't seem right since after saving the FileStream, I can open testbmp2.bmp in Photoshop and see the alpha channel.
Update 2:
I think I need to re-word the issue I'm having. I'm trying to display separate channels of a texture. I am displaying the bitmap via an Image control. I have a simple HLSL pre compiled shader assigned to the Image's Effect property that will mask out the channels based user input. Not being able to get the BitmapSource into a BitmapImage while retaining the alpha channel was only part of the problem. I realize now that since my original BitmapSource's Format was Bgra32 I could assign that directly to the Image's Source property. The problem appears to be that an Image object will only display texels with pre-multiplied alpha...? Here's my shader code:
sampler2D inputImage : register(s0);
float4 channelMasks : register(c0);
float4 main (float2 uv : TEXCOORD) : COLOR0
{
float4 outCol = tex2D(inputImage, uv);
if (!any(channelMasks.rgb - float3(1, 0, 0)))
{
outCol.rgb = float3(outCol.r, outCol.r, outCol.r);
}
else if (!any(channelMasks.rgb - float3(0, 1, 0)))
{
outCol.rgb = float3(outCol.g, outCol.g, outCol.g);
}
else if (!any(channelMasks.rgb - float3(0, 0, 1)))
{
outCol.rgb = float3(outCol.b, outCol.b, outCol.b);
}
else
{
outCol *= channelMasks;
}
if (channelMasks.a == 1.0)
{
outCol.r = outCol.a; // * 0.5 + 0.5;
outCol.g = outCol.a;
outCol.b = outCol.a;
}
outCol.a = 1;
return outCol;
}
I've tested it pretty extensively and I'm pretty sure that its setting the outCol value properly. When i pass in a channel mask value of (1.0, 1.0, 1.0, 0.0) the image displayed is the RGB channels with the areas of the alpha that are black drawing as black - as you would expect if WPF was pre-multiplying the alpha on to the RGB channels. Does anyone know of any way to dispaly a BitmapSource in an Image without pre multiplied alpha? Or more to the point, is there a way to have the effect receive the texture before the alpha pre-multiply happens? I'm not sure what order things are done in, but apparently the pre-multiply happens before the texture is written to the s0 register and the shader gets to work on it. I could try to do this with WriteableBitmap and copying the alpha out to another BitmapSource but that would all be using software rendering i think...?
I ran into a similar problem lately. The BmpBitmapEncoder does not care about the alpha channel and therefore does not preserve it. An easy fix is to use a PngBitmapEncoder instead.
I'm trying to display the separate R, G, B and A channels of a texture based on user input. I'm using an Image class to display textures that have alpha channels. These textures are loaded in to BitmapSource objects and have a Format of Bgra32. The problem is that when I set the Image's Source to the BitmapSource, if I display any combination of the R, G or B channels, I always get pixels that are pre-multiplied by the alpha value. I wrote a really simple shader, pre-compiled it, and used a ShaderEffect class to assign to the Image's Effect property in order to separate and display the separate channels, but apparently, the shader is given the texture after WPF has pre-multiplied the alpha value onto the texture.
Here's the code snippet for setting the Image's Source:
BitmapSource b = MyClass.GetBitmapSource(filepath);
// just as a test, write out the bitmap to file to see if there's an alpha.
// see attached image1
BmpBitmapEncoder test = new BmpBitmapEncoder();
//test.Compression = TiffCompressOption.None;
FileStream stest = new FileStream(#"c:\temp\testbmp2.bmp", FileMode.Create);
test.Frames.Add(BitmapFrame.Create(b));
test.Save(stest);
stest.Close();
// effect is a class derived from ShaderEffect. The pixel shader associated with the
// effect displays the color channel of the texture loaded in the Image object
// depending on which member of the Point4D is set. In this case, we are showing
// the RGB channels but no alpha
effect.PixelMask = new System.Windows.Media.Media3D.Point4D(1.0f, 1.0f, 1.0f, 0.0f);
this.image1.Effect = effect;
this.image1.Source = b;
this.image1.UpdateLayout();
Here's the shader code (its pretty simple, but I figured I'd include it just for completeness):
sampler2D inputImage : register(s0);
float4 channelMasks : register(c0);
float4 main (float2 uv : TEXCOORD) : COLOR0
{
float4 outCol = tex2D(inputImage, uv);
if (!any(channelMasks.rgb - float3(1, 0, 0)))
{
outCol.rgb = float3(outCol.r, outCol.r, outCol.r);
}
else if (!any(channelMasks.rgb - float3(0, 1, 0)))
{
outCol.rgb = float3(outCol.g, outCol.g, outCol.g);
}
else if (!any(channelMasks.rgb - float3(0, 0, 1)))
{
outCol.rgb = float3(outCol.b, outCol.b, outCol.b);
}
else
{
outCol *= channelMasks;
}
if (channelMasks.a == 1.0)
{
outCol.r = outCol.a;
outCol.g = outCol.a;
outCol.b = outCol.a;
}
outCol.a = 1;
return outCol;
}
Here's the output from the code above:
(sorry, i don't have enough reputation points to post images or apparently more than 2 links)
The file save to disk (C:\temp\testbmp2.bmp) in photoshop:
http://screencast.com/t/eeEr5kGgPukz
Image as displayed in my WPF application (using image mask in code snippet above):
http://screencast.com/t/zkK0U5I7P7
I am trying to get an HLSL Pixel Shader for Silverlight to work to subtract the background image from a video image. Can anyone suggest a more sophisticated algorithm than I am using because my algorithm isn't doing it correctly?
float Tolerance : register(C1);
SamplerState ImageSampler : register(S0);
SamplerState BackgroundSampler : register(S1);
struct VS_INPUT
{
float4 Position : POSITION;
float4 Diffuse : COLOR0;
float2 UV0 : TEXCOORD0;
float2 UV1 : TEXCOORD1;
};
struct VS_OUTPUT
{
float4 Position : POSITION;
float4 Color : COLOR0;
float2 UV : TEXCOORD0;
};
float4 PS( VS_OUTPUT input ) : SV_Target
{
float4 color = tex2D( ImageSampler, input.UV );
float4 background = tex2D( BackgroundSampler, input.UV);
if (abs(background.r - color.r) <= Tolerance &&
abs(background.g - color.g) <= Tolerance &&
abs(background.b - color.b) <= Tolerance)
{
color.rgba = 0;
}
return color;
}
To see an example of this, you need a computer with a webcam:
Go to the page http://xmldocs.net/alphavideo/background.html
Press [Start Recording].
Move your body out of the the scene and press [Capture Background].
Then move your body back into the scene and use the slider to adjust the Toleance value to the shader.
EDIT
Single pixel isn't useful for such task, because of noise. So algorithm essence should be to measure similarity between pixel blocks. Recipe pseudo-code (based on correlation measurement):
Divide image into N x M grid
For each N,M cell in grid:
correlation = correlation_between(signal_pixels_of(N,M),
background_pixels_of(N,M)
);
if (correlation > threshold)
show_background_cell(N,M)
else
show_signal_cell(N,M)
This is sequential pseudo code, but it could be easily converted to HLSL shader. Simply each pixel detects to which pixel block it belongs, and after that measures correlation between corresponding blocks. And based on that correlation shows or hides current pixel.
Try this approach,
Good Luck !