Using the FFmpeg C API I have encoding and decoding a video working. However, the re-encoded video stream does not maintain the original video's orientation (rotation). So vertical videos have been flipped horizontal.
I’m not sure how to resolve this. Is there a metadata field that gets set? Using MediaInfo I see the original video has a metadata field ’Rotation : 90°’ and the new video does not.
Or does each encoded frame need to be rotated vertically?
I’ve looked at the decode frame's side_data but it is empty.
for (j = 0; j < decoded_frame->nb_side_data; j++) {
AVFrameSideData *sd = decoded_frame->side_data[j];
if(sd->type == AV_FRAME_DATA_DISPLAYMATRIX) {
LOGI("=> displaymatrix: rotation of %.2f degrees", av_display_rotation_get((int32_t *)sd->data));
}
}
I resolved this by adding 'Rotation' to the output video stream's metadata.
av_dict_copy(&output_stream->metadata, input_stream->metadata, AV_DICT_DONT_OVERWRITE);
There is a good explanation of the rotation metadata field here:
Correct Smartphone Video Orientation and How To Rotate iOS and Android Videos with ffmpeg
Related
I'm using the H.264 library to compress a video frame by frame. It works, I can replay it back locally without any issue.
However, I need to send that video over the LAN and that LAN is rather busy already so I need to limit the size of each frame to a maximum of about 250Kb.
I use the following code to setup the parameters, but changing the bit rate values does not seem to have any effect on what the library does with the input frames:
x264_param_t param = {};
if(x264_param_default_preset(¶m, "faster", nullptr) < 0)
{
return -1;
}
param.i_csp = X264_CSP_I420;
param.i_width = 3840;
param.i_height = 2160;
param.i_keyint_max = static_cast<int>(f_frame_header.f_fps);
param.i_threads = X264_THREADS_AUTO;
param.b_vfr_input = 0;
param.b_repeat_headers = 1;
param.b_annexb = 1;
// the following three parameters are the ones I tried to change with no results
param.rc.i_bitrate = 100000;
param.rc.i_vbv_max_bitrate = 100000;
param.rc.i_vbv_buffer_size = 125000;
if(x264_param_apply_profile(¶m, "high") < 0)
{
return -1;
}
...enter loop reading frames and compressing them...
Changing the i_bitrate, i_vbv_max_bitrate and i_vbv_buffer_size parameters seems to have absolutely no effect on the size of the resulting frames. I still get some frames over 500Kb and in many even, rather large frames one after the other as the following sizes show:
20264
358875
218429
20728
25215
310230
36127
9077
29785
341541
222778
23542
21356
276772
25339
32459
421036
11179
6172
286070
193849
What I would need is the largest frame to be around 250,000 at its maximum. Now I understand that once in a while it go over a bit, but not 2×. That's just too much for my current available bandwidth.
What am I doing wrong in the parameters setup above?
I've seen this command line:
ffmpeg -i input -c:v libx264 -b:v 2M -maxrate 2M -bufsize 1M output.mp4
which would suggest that what I'm doing above should work (I tried all sorts of values including the ones one that command line). Yet the frame size does not really change between my runs.
I tried with a blur applied to each frame to see whether it work help. Yes! It did. The result is a movie which is 2.44 times smaller than the original.
To load each JPEG image from the original, I use ImageMagick++ (in C++), so I just do the following blur on each image:
image.blur(0.0, 5.0);
and that took about 10 hours total (without the blur the same processing took about 40 minutes) but it was worth it since in the end the compressed movie went from 1,293,272,023 bytes to only 529,556,265 bytes (2.44218 times smaller). The blur added about 3.3 seconds of processing per frame and there are a little over 11,000 frames in the original.
Note: I used 5.0 for the blur because I have 4K images and although I can see a sharp difference when I look at one frame, when playing back the resulting movie, I don't notice the final blur. If you have smaller images, you probably want to use a smaller number. It looks like many people use a blur of just 0.05 and already have good results in compression ratios.
In C, use the BlurImage() function:
Image *BlurImage(const Image *image,const double radius,
const double sigma,ExceptionInfo *exception)
Here are some references about using a blur to further compress JPEG images as it helps eliminates sharp edges which do not compress well in the JPEG format (as sharp edge are not as natural):
Recommendation for compressing JPG files with ImageMagick
How do I reduce the file size of an image? (search on "blur" to find the section)
Could I blur an image to dramatically reduce the file size?
There is a very good chance that I am going down a pointless path on this, so I apologize if this is a waste of time. I have been trying to write uncompressed video to an FLV file, and I am not sure whether it is possible.
According to Wikipedia, a valid video encoding option is 0, which indicates an "RGB" video encoding: https://en.wikipedia.org/wiki/Flash_Video#Packets. However, I don't see any mention of this Codec ID option in Adobe's documentation; neither "Video File Format Specification Version 10" nor "Adobe Flash Video File Format Specification Version 10.1".
I proceeded under the assumption that a 0/RGB Codec ID is allowed. I hard-coded an array of unsigned char in C and used fwrite to write the following Double/Number metadata to a new, binary FLV file (which admittedly, I am assuming I wrote correctly):
duration: 4 (seconds)
width: 16 (pixels)
height: 16 (pixels)
videodatarate: 6 (Kbps)
framerate: 1 (fps)
videocodecid: 0
filesize: 3323 (bytes)
I then added 4 VIDEODATA tags, 1 for each RGB frame I was hoping to write. Their timestamps are 0, 1000, 2000, and 3000 (milliseconds). All four of them have a 769-byte payload: the first byte to specify it is a keyframe with a Codec ID of 0, and the remaining 768 are to represent a 16x16x3 (RGB) image. I wrote 255/0xFF for all values in hopes of seeing a small, white screen appear for 4 seconds.
When that did not play correctly in VLC Media Player, as I feared, I tried using RGBA colors for each frame. I also changed the videodatarate and filesize metadata to Number values 8 (Kbps) and 4347 (bytes) respectively.
Unfortunately, this did not play in VLC Media Player either. I was wondering if anyone knew for certain whether uncompressed video in an FLV file is possible? If so, I was curious what format the video data should be in (RGB, RGBA, multiple VIDEODATA tags, just one VIDEODATA tag, etc.)?
My C code is mostly one, giant array of unsigned char, but if anyone would like to see it, I can try adding it. Any advice is greatly appreciated.
Thank you,
Mitchell A
As per SirDarius, "the video encoding types listed in the Wikipedia page do not come from an official source. I would not recommend relying on those." This makes sense given that the FLV Format documentation from Adobe itself makes no mention of an uncompressed, RGB option for video encoding.
I was holding out hope that Wikipedia editors and other people knew of some undocumented easter egg in the FLV format, but I'm now convinced that's not the case.
"...The FLV Format documentation from Adobe itself makes no mention of an uncompressed, RGB option for video encoding."
For RGB (raw bitmap data) you must use theScreen 1 codec (id=3).
Strangely, it's hidden in the SWF Format documentation (not the FLV Format docs).
See Chapter 14 (page 204) which is the Video section...
You want specifically page 208 for the Screen Video codec to be explained.
Check this example code (AS3) of encoding RGB into Screen Video.
Apply the logic, especially function videoData(), which could be adjusted to read pixels uints (via some getPixel type call) or just read from an Array.
Example:
for (var x2:int = 0; x2 < xLimit; x2++)
{
var px:int = (x1 * blockWidth) + x2;
var py:int = frameHeight - ((y1 * blockHeight) + y2); // (flv's save image from bottom to top)
var p:uint = YOUR_INPUT_BITMAP.getPixel(px, py); // sample a pixel's RGB (3-bytes unsigned int)
//# IF reading from Pixel's uint value
block.writeByte( p & 0xff ); // blue
block.writeByte( p >> 8 & 0xff ); // green
block.writeByte( p >> 16 ); // red
//# ELSE IF reading from Array of R-G-B values(FLV writes in BGR format)
block.writeByte( myRGB_Array[x+2] ); // blue
block.writeByte( myRGB_Array[x+1] ); // green
block.writeByte( myRGB_Array[x] ); // red
}
What would be the best way, in linux from gnu C and not C++, to display a gif87a file on screen and redisplay it in the same location on the screen so the user can observe changes that are made on the fly to the dataset? This is not an animated gif.
in some old code (fortran77) that has a C wrapper which takes an image that was displayed on the screen and writes it to a gif file, there is a comment about X Window Applications Programming, Ed. 2, Johnson & Reichard that was used as a reference to write the C code to display image data to the screen and write a gif87a file, and this code was written around 1995, the onscreen display of the image no longer works (just a black window) but the creation of the gif file still works. What i would like to do is from the existing C code, in SLES version 11.4 with the libraries that are available to open the gif file and display it on screen. The image, or contour plot, has a color bar that the user sets the min/max value for to display the image to their liking and it would be preferable to make it as easy & efficient for the user to adjust those min max values then redraw the image (re-write the gif then redisplay on screen in same location). There's also a handful of other knobs that the user can turn, such as windowing of the dat (hamming or han) and it would be best if the user can quickly/easily run though about 5+ ways of looking at the image before settling on what is considered correct then using that final gif that was created in powerpoint, excel, etc.
Writing an X11 application is non-trivial. You can display a GIF (or any one of around 200 image formats) using ImageMagick which is included in most Linux distros and is available for macOS. Windows doesn't count.
So, you can create images and manipulate images from the command line, or in C if you want. So, let's create a GIF that is 1024x768 and full of random colours:
convert -size 1024x768 xc:blue +noise random -pointsize 72 -gravity center -annotate 0 "10" image.gif
Now we can display it, using ImageMagick's display program:
display image.gif &
Now we can get its X11 "window-id" with:
xprop -root
...
_NET_ACTIVE_WINDOW(WINDOW): window id # 0x600011
...
...
Now you can change the image, however you like with filters and blurs and morphology and thresholds and convolutions:
convert image.gif -threshold 80% -morphology erode diamond -blur 0x3 -convolve "3x3: -1,0,1, -2,0,2, -1,0,1" ... image.gif
And then tell the display program to redraw the window with:
display -window 0x600011 image.gif
Here is a little script that generates images with a new number in the middle of each frame and updates the screen:
for ((t=0;t<100;t++)) ; do
convert -size 640x480 xc:blue +noise random -pointsize 72 -fill white -gravity center -annotate 0 "$t" image.gif
display -window 0x600011 image.gif
done
Now all you need to do is find a little Python or Tcl/Tk library that draws some knobs and dials, reads their positions and changes the image accordingly and tells the screen to redraw.
As a result of the lack of enthusiasm for my other answer, I thought I'd have another attempt. I had a quick look and learn of Processing which is a very simple language, very similar to C but much easier to program.
Here is a screen shot of it loading a GIF and displaying a couple of twiddly knobs - one of which I attached to do a threshold on the image.
Here's the code - it is not the prettiest in the world because it is my first ever code in Processing but you should be able to see what it is doing and adapt to your needs:
import controlP5.*;
ControlP5 cp5;
int myColorBackground = color(0,0,0);
int knobValue = 100;
float threshold=128;
Knob myKnobA;
Knob myKnobB;
PImage src,dst; // Declare a variable of type PImage
void setup() {
size(800,900);
// Make a new instance of a PImage by loading an image file
src = loadImage("image.gif");
// The destination image is created as a blank image the same size as the source.
dst = createImage(src.width, src.height, RGB);
smooth();
noStroke();
cp5 = new ControlP5(this);
myKnobA = cp5.addKnob("some knob")
.setRange(0,255)
.setValue(50)
.setPosition(130,650)
.setRadius(100)
.setDragDirection(Knob.VERTICAL)
;
myKnobB = cp5.addKnob("threshold")
.setRange(0,255)
.setValue(220)
.setPosition(460,650)
.setRadius(100)
.setNumberOfTickMarks(10)
.setTickMarkLength(4)
.snapToTickMarks(true)
.setColorForeground(color(255))
.setColorBackground(color(0, 160, 100))
.setColorActive(color(255,255,0))
.setDragDirection(Knob.HORIZONTAL)
;
}
void draw() {
background(0);
src.loadPixels();
dst.loadPixels();
for (int x = 0; x < src.width; x++) {
for (int y = 0; y < src.height; y++ ) {
int loc = x + y*src.width;
// Test the brightness against the threshold
if (brightness(src.pixels[loc]) > threshold) {
dst.pixels[loc] = color(255); // White
} else {
dst.pixels[loc] = color(0); // Black
}
}
}
// We changed the pixels in destination
dst.updatePixels();
// Display the destination
image(dst,100,80);
}
void knob(int theValue) {
threshold = color(theValue);
println("a knob event. setting background to "+theValue);
}
void keyPressed() {
switch(key) {
case('1'):myKnobA.setValue(180);break;
case('2'):myKnobB.setConstrained(false).hideTickMarks().snapToTickMarks(false);break;
case('3'):myKnobA.shuffle();myKnobB.shuffle();break;
}
}
Here are some links I used - image processing, P5 library of widgets and knobs.
I have created a sample using grafika code where I am creating the video from camera feeds. In grafika sample there is a method to drawExtra while passing data to muxer which add dots in vidoe. I want to add water mark in place of this at left top corner.
Please suggest.
if (mFileSaveInProgress && showCam)
{
Log.e(TAG, "drawFrame saving to new video");
mEncoderSurface.makeCurrent();
GLES20.glViewport(0, 0,720, 1280);
mFullFrameBlit.drawFrame(mTextureId, mTmpMatrix);
drawExtra(mFrameNum, viewWidth, viewHeight);
mCircEncoder.frameAvailableSoon();
mEncoderSurface.setPresentationTime(mvideoTexture.getTimestamp());
mEncoderSurface.swapBuffers();
}
Our tool allows export to PNG, which works very nicely.
Now, I would like to add export to some vector format. I tried XPS, but the results are not satisfying at all.
Take a look at a comparison http://www.jakubmaly.cz/xps-vs-png.png.
The picture on the left comes from an XPS export, the picture on the right from PNG export, the XPS picture is visibly blurred when opened in XPS Viewer and zoomed 100%.
Are there any settings that I am missing or why is it so?
Thanks,
Jakub.
A sample xps output can be found here: http://www.jakubmaly.cz/files/a.xps.
This is the code that does the XPS export:
if (!boundingRectangle.HasValue)
{
boundingRectangle = new Rect(0, 0, frameworkElement.ActualWidth, frameworkElement.ActualHeight);
}
// Save current canvas transorm
Transform transform = frameworkElement.LayoutTransform;
// Temporarily reset the layout transform before saving
frameworkElement.LayoutTransform = null;
// Get the size of the canvas
Size size = new Size(boundingRectangle.Value.Width, boundingRectangle.Value.Height);
// Measure and arrange elements
frameworkElement.Measure(size);
frameworkElement.Arrange(new Rect(size));
// Open new package
System.IO.Packaging.Package package = System.IO.Packaging.Package.Open(filename, FileMode.Create);
// Create new xps document based on the package opened
XpsDocument doc = new XpsDocument(package);
// Create an instance of XpsDocumentWriter for the document
XpsDocumentWriter writer = XpsDocument.CreateXpsDocumentWriter(doc);
// Write the canvas (as Visual) to the document
writer.Write(frameworkElement);
// Close document
doc.Close();
// Close package
package.Close();
// Restore previously saved layout
frameworkElement.LayoutTransform = transform;
Interesting (and annoying) issue - you may want to check out the lengthy answer from Jo0815 to Printing XpsDocument causes resampled images (96dpi?) - FixedDocument prints sharp, quoting a Microsoft support response - a couple of excerpts:
Some vector features from WPF cannot be emulated in our GDI code and
we resort to converting subsets of the scene to GDI bitmaps. These
bitmaps are the cause of the blurred zooming.
[...]
These bitmaps are the cause of the blurred zooming. The problem is
that the WPF is being rasterised to a bitmap at the -wrong resolution.
The print path is designed to rasterise unsupported features into a
bitmap, but it is supposed to do it at device resolution. Instead the
rasterisation is always being done at 96dpi. That's fine for a screen
but produces blurred output for a 600dpi printer. [emphasis mine]
Please note that the latter will apply for nowadays higher DPI screens as well of course, I've encountered blurring like this various times already - do you by chance use a high DPI monitor?
Now, apparently Microsoft is not entirely in control of the apparatus regarding this:
Additionally the problem only occurs when printing XPS and isn't a
problem when printing XAML directly. I'm pretty sure there is
documentation somewhere that says XPS will print at device resolution.
[...] It is something we
plan to improve in the next version of the product but not for Win 7.
The problem is that when printing XAML it will correctly render the
image at 600dpi, but when printing XPS it will still render the image
at 96dpi. Since XAML is converted to XPS before printing it seems
highly odd that one method of printing XPS produces different results
to another method of printing XPS. [emphasis mine]
[...]
There is no UI to configure the XPS Document Writer DPI. If you later
print a generated XPS document at a different DPI from the writers
internal default you may get poor results for bitmap content. With GDI
printers you can control the final DPI and your final desitination is
usally paper - no chance to reprint the document.
Conclusion
In conclusion, I'd still try to adjust PrintTicket.PageResolution Property within Néstor Sánchez' approach (+1), if your use case does allow this (though I remotely recall reading somewhere, that this doesn't have any effect as well); section Bitmap Resolution and Pixel Format in Using the XPS Rasterization Service confirms the issue he encountered with FixedDocument:
XPS rasterizer object for a fixed page must know the resolution at
which the page will be rendered. The XPSDrv filter specifies this
resolution, in dots per inch (DPI), as an input parameter [...] For example, if a display device has a resolution
of 600 DPI, and a fixed page describes a standard letter-size page, a
bitmap image of the entire page has the following dimensions [...]
Workaround
As a potential workaround you might want to explore alexandrud's solution for the related question How to convert a XPS file to an image in high quality (rather than blurry low resolution)?, which recommends using xps2img, a XPS (XML Paper Specification) document to set of images conversion utility. In particular it Allows to specify images size or DPI, which might help depending on the print path solution applied in turn.
Good luck!
I've had a similar problem. My image was very blurry when passed to XPS intermediated thru a FixedDocument.
The solution was to write the image directly to the XPS...
/// <summary>
/// Saves the supplied visual Source, within the specified Bounds, as XPS in the specified File-Name.
/// Returns error message or null when succeeded.
/// </summary>
public static string SaveVisualAsXPS(Visual Source, Size Bounds, string FileName)
{
string ErrorMessage = null;
try
{
using (var Container = Package.Open(FileName, FileMode.Create))
{
using (var TargetDocument = new XpsDocument(Container, CompressionOption.Maximum))
{
var Writer = XpsDocument.CreateXpsDocumentWriter(TargetDocument);
var Ticket = GetPrintTicketFromPrinter();
if (Ticket == null)
return "No printer is defined.";
Ticket.PageMediaSize = new PageMediaSize(Bounds.Width, Bounds.Height);
var SourceVisual = Source;
Writer.Write(SourceVisual, Ticket);
}
}
}
catch (Exception Problem)
{
ErrorMessage = "Cannot export document to XPS.\nProblem: " + Problem.Message;
}
return ErrorMessage;
}
Giving a print-ticket with the exact width and height avoids scaling (that was I wanted in my case).
Get the function from the example in:
http://msdn.microsoft.com/en-us/library/system.printing.printticket.aspx