List available capture formats - c

New to V4L, I decided to start using the video4linux2 library in order to capture a frame from my camera in C (I am using the uvcvideo module with a Ricoh Co. camera). I followed several guides and tutorials, and managed to get a running program. My question is mainly about this usual code line :
struct v4l2_format format = {0};
format.fmt.pix.pixelformat = V4L2_PIX_FMT_MJPEG;
// ...
This is where I set the actual video format to use when capturing. As you can see, in this sample, I'm using MJPEG (http://lxr.free-electrons.com/source/include/uapi/linux/videodev2.h#L390). Even though this might be a great format and all, my application will probably require simple RGB formatting, pixel per pixel I guess. For this reason, I tried using RGB format constants such as V4L2_PIX_FMT_RGB24. Now for some reason... v4l2 doesn't like it. I'm guessing this is hardware-related, but I'd like to avoid MJPEG manipulations as much as possible. For testing purposes, I tried using other constants and formats, but no matter what I did, v4l2 kept changing the pixelformat field's value :
xioctl(fd, VIDIOC_S_FMT, &format); // This call succeeds with errno != EINTR.
if(format.fmt.pix.pixelformat != V4L2_PIX_FMT_RGB24){
// My program always enters this block when not using MJPEG.
fprintf(stderr, "Format wasn't accepted by v4l2.");
exit(4);
}
Now my question is : is there a way I could get a list of accepted video formats (and I mean, accepted by my camera/v4l2), from which I could pick something else than MJPEG ? If you think I have to stick with MJPEG, would you recommend me any library allowing me to manipulate it, and eventually, to draw back the capture in a GUI frame ?
Barbarian test code
I used the following trick to test all available formats on my hardware. First, some shell script to get a list of all formats...
grep 'V4L2_PIX_FMT' /usr/include/linux/videodev2.h | grep define | tr '\t' ' ' | cut -d' ' -f2 | sed 's/$/,/g'
... the output of which is used in this C program :
int formats[] = {/* result of above command */};
int i = 0;
struct v4l2_format format = {0};
for(i = 0; i < /* number of lines in previous command output */; i++){
format.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
format.fmt.pix.width = 320;
format.fmt.pix.height = 240;
format.fmt.pix.pixelformat = formats[i];
format.fmt.pix.field = V4L2_FIELD_NONE;
if(xioctl(fd, VIDIOC_S_FMT, &format) != -1 && format.fmt.pix.pixelformat == formats[i])
fprintf(stderr, "Accepted : %d\n", i);
}
This test reveals that only V4L2_PIX_FMT_YUYV and V4L2_PIX_FMT_MJPEG are functional. Any way I could improve this, or is it hardware-related ?

In Linux, command line utility v4l2-ctl displays all of a webcam's natively supported formats -- install it with sudo apt-get install v4l-utils, run it with v4l2-ctl -dX --list-formats-ext where X is the camera index as in /dev/videoX. These formats are reported to the v4l2 kernel module via uvcvideo module and they are supported natively by the webcam chipset. Only the listed formats are supported by v4l2, anything else would need to be coded by the user, and RGBs are very seldom provided, despite virtually all CCDs working in Bayer RGGB. The most common formats by far are YUV422 (YUYV or YUY2) and MJPEG, with a certain overlap: MJPEG achieve larger frame rates for large resolutions.
C++ code for listing the camera formats can be found in Chromium GetDeviceSupportedFormats() implementation for Linux here.
If you have to plug code to convert YUV to RGB I'd recommend libyuv which has been optimized for plenty of architectures.

In order to enumerate the available format you can use ioctl VIDIOC_ENUM_FMT
To print descriptions of capture format supported by /dev/video0 you can process like this :
#include <string.h>
#include <stdio.h>
#include <fcntl.h>
#include <linux/videodev2.h>
int main()
{
int fd = v4l2_open("/dev/video0", O_RDWR);
if (fd != -1)
{
struct v4l2_fmtdesc fmtdesc;
memset(&fmtdesc,0,sizeof(fmtdesc));
fmtdesc.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
while (ioctl(fd,VIDIOC_ENUM_FMT,&fmtdesc) == 0)
{
printf("%s\n", fmtdesc.description);
fmtdesc.index++;
}
v4l2_close(fd);
}
}

Related

X264 : How to access NAL units from encoder?

When I call
frame_size = x264_encoder_encode(encoder, &nals, &i_nals, &pic_in, &pic_out);
and subsequently write each NAL to a file like this:
if (frame_size >= 0)
{
int i;
int j;
for (i = 0; i < i_nals; i++)
{
printf("******************* NAL %d (%d bytes) *******************\n", i, nals[i].i_payload);
fwrite(&(nals[i].p_payload[0]), 1, nals[i].i_payload, fid);
}
}
then I get this
My questions are:
1) Is it normal that there's readable parameters in the beginning of the file?
2) How do I configure the X264 encoder so that the encoder returns frames that I can send via UDP without the packet getting fragmented (size must be below 1390 or somewhere around that).
3) With the x264.exe I pass in these options:
"--threads 1 --profile baseline --level 3.2 --preset ultrafast --bframes 0 --force-cfr --no-mbtree --sync-lookahead 0 --rc-lookahead 0 --keyint 1000 --intra-refresh"
How do I map those to the settings in the X264 parameters structure ? (x264_param_t)
4) I have been told that the x264 static library doesn't support bitmap input to the encoder and that I have to use libswscale for conversion of the 24bit RGB input bitmap to YUV2. The encoder, supposedly, only takes YUV2 as input? Is this true? If so, how do I build libswscale for the x264 static library?
1) Yes. x264 includes the automatically. Its an SEI slice, and you can throw it away if you want.
2) set i_slice_max_size = 1390
3) Take a look at x264_param_t in x264.h. The settings are fairly self explanatory. As for setting the profile and preset call int x264_param_apply_profile( x264_param_t *, const char *profile ) and int x264_param_default_preset( x264_param_t *, const char *preset, const char *tune )
4) Yes, it is true, I want lying when I said that. Look online/on stack overflow there are a million resources on compiling ffmpeg. In fact if you compiled x264 with avcodec support you already have it on your system.
5) Yes!, you should be a good stack overflow citizen and up vote and accept answers form people who donate there free time and knowledge (which takes years to acquire) to helping you.

How can I seek to frame No. X with ffmpeg?

I'm writing a video editor, and I need to seek to exact frame, knowing the frame number.
Other posts on stackoverflow told me that ffmpeg may give me a few broken frames after seeking, which is not a problem for playback but a big problem for video editors.
And I need to seek by frame number, not by time, which will become inaccurate when converted to frame number.
I've read dranger's tuts (which is outdated now), and end up with:
av_seek_frame(fmt_ctx, video_stream_id, frame, AVSEEK_FLAG_ANY);
It always seek to frame No. 0, and always return 0 which means success.
Then I tried to read Blender's source code and found it really complex(maybe I should implement an image buffer?).
So, is there any simple way to seek to a frame with just a simple call like seek(context, frame_number)(while getting a full frame, not a broken one)? Or, is there any lightweight library that simplifies this?
EDIT:
Thanks to praks411,I found the solution:
void AV_seek(AV * av, size_t frame)
{
int frame_delta = frame - av->frame_id;
if (frame_delta < 0 || frame_delta > 5)
av_seek_frame(av->fmt_ctx, av->video_stream_id,
frame, AVSEEK_FLAG_BACKWARD);
while (av->frame_id != frame)
AV_read_frame(av);
}
void AV_read_frame(AV * av)
{
AVPacket packet;
int frame_done;
while (av_read_frame(av->fmt_ctx, &packet) >= 0) {
if (packet.stream_index == av->video_stream_id) {
avcodec_decode_video2(av->codec_ctx, av->frame, &frame_done, &packet);
if (frame_done) {
...
av->frame_id = packet.dts;
av_free_packet(&packet);
return;
}
}
av_free_packet(&packet);
}
}
EDIT2:
Turns out there is a library for this: FFMS2.
It is "an FFmpeg based source library [...] for easy frame accurate access", and is portable (at least across Windows and Linux).
av_seek_frame will only seek based on timestamp to the key-frame. Since it seeks to the keyframe, you may not get what you want. Hence it is recommended to seek to nearest keyframe and then read frame by frame util you reach the desired frame.
However, if you are dealing with fixed FPS value, then you can easily map timestamp to frame index.
Before seeking you will need to convert your time to AVStream.time_base units if you have specified stream. Read ffmpeg documentation of av_seek_frame in avformat.h.
For example, if you want to seek to 1.23 seconds of clip:
double m_out_start_time = 1.23;
int flgs = AVSEEK_FLAG_ANY;
int seek_ts = (m_out_start_time*(m_in_vid_strm->time_base.den))/(m_in_vid_strm->time_base.num);
if(av_seek_frame(m_informat, m_in_vid_strm_idx,seek_ts, flgs) < 0)
{
PRINT_MSG("Failed to seek Video ")
}

invalid argument error when setting yres_virtual in fb_var_screeninfo

I'm trying to make an application for linux that writes directly to the framebuffer /dev/fb0. In order to make it double buffered I try to make the virtual screen be double the size of the screen. This is the program I wrote:
struct fb_var_screeninfo screeninfo_var;
struct fb_fix_screeninfo screeninfo_fixed;
unsigned int* screenbuffer;
void gfx_init()
{
fb0 = open("/dev/fb0", O_RDWR);
if(fb0 == 0)
error("Could not open framebuffer located in /dev/fb0!");
if (ioctl(fb0, FBIOGET_FSCREENINFO, &screeninfo_fixed) == -1)
error("Could not retrive fixed screen info!");
if (ioctl(fb0, FBIOGET_VSCREENINFO, &screeninfo_var) == -1)
error("Could not retrive variable screen info!");
screeninfo_var.xres_virtual = screeninfo_var.xres;
screeninfo_var.yres_virtual = screeninfo_var.yres * 2;
screeninfo_var.width = screeninfo_var.xres;
screeninfo_var.height = screeninfo_var.yres;
screeninfo_var.xoffset = 0;
screeninfo_var.yoffset = 0;
if (ioctl(fb0, FBIOPUT_VSCREENINFO, &screeninfo_var) == -1)
error("Could not set variable screen info!");
info("Detected monitor of %ix%i pixels using %i bit colors.",screeninfo_var.xres, screeninfo_var.yres, screeninfo_var.bits_per_pixel);
screenbuffersize = screeninfo_var.xres_virtual * screeninfo_var.yres_virtual * screeninfo_var.bits_per_pixel/8;
screenbuffer = (unsigned int *)mmap(0, screenbuffersize, PROT_READ | PROT_WRITE, MAP_SHARED, fb0, 0);
if( (long)screenbuffer == 0 || (long)screenbuffer == -1 )
error("Failed to map framebuffer to device memory!");
}
The program failes on ioctl(fb0, FBIOPUT_VSCREENINFO, &screeninfo_var) reporting the error invalid argument. When removing the line screeninfo_var.yres_virtual = screeninfo_var.yres * 2; it runs fine (but without double buffering).
Does someone see what I do wrong here?
To save anyone a headache in the future, there is a way to properly double buffer using low level graphics on linux (such as /dev/fb0). However per this thread: https://forum.odroid.com/viewtopic.php?f=55&t=8741 it is not possible to truly double buffer the framebuffer by creating a virtual framebuffer of double the size of the original one (I've read that the raspberry PI might be an exception to this rule because it is backed by a different driver).
The right way to double buffer using low level graphics on linux is to go through libdrm (or /dev/dri/card0). There is a really nice example that I followed myself here: https://github.com/dvdhrm/docs/blob/master/drm-howto/modeset-vsync.c. It wasn't very hard to convert my /dev/fb0 code to /dev/dri/card0 code.
Anyways I hope I've helped someone out here who might come across this in the future.
This is a common question and it is caused by the limitation of the video driver. E.g., Intel's 810 chipset only allows 640x480 resolution and the Broadcom's limit the width to no more than 1200 (http://www.raspberrypi.org/phpBB3/viewtopic.php?f=66&t=29968).
Not much you can do here, unless you change the driver or the videocard itself (if possible).
EDIT: if you are on a PC, try using the vesafb, not the intel's driver.
There's a hint here: http://www.mail-archive.com/debian-russian#lists.debian.org/msg27725.html

wsdisplay color map not being retrieved correctly

I've been trying to use wscons and wsdisplay on NetBSD 5.1.2 (using the VESA framebuffer implementation) recently and I've encountered a bit of a problem:
I can set color maps successfully and they look correct but getting color maps seems to return incorrect data, such that when I try to restore the original color map once the program has finished, all the colors are incorrect:
Here's a reduced program causing the problem (note that it must be run either as root or as the user logged in on the second virtual terminal (/dev/ttyE1)):
#include <stdint.h>
#include <stdlib.h>
#include <stdio.h>
#include <fcntl.h>
#include <sys/ioctl.h>
#include <unistd.h>
#include <dev/wscons/wsconsio.h>
int main(int argc, char **argv) {
(void)argc, (void)argv;
int tty = open("/dev/ttyE1", O_RDWR | O_EXCL);
if(tty == -1) {
perror("error opening tty");
return EXIT_FAILURE;
}
struct wsdisplay_fbinfo fbinfo;
if(ioctl(tty, WSDISPLAYIO_GINFO, &fbinfo) == -1) {
perror("error retrieving framebuffer info");
close(tty);
return EXIT_FAILURE;
}
uint8_t *cmap_data = malloc(fbinfo.cmsize * 3);
if(cmap_data == NULL) {
perror("error allocating memory for color map data");
close(tty);
return EXIT_FAILURE;
}
struct wsdisplay_cmap cmap;
cmap.index = 0;
cmap.count = fbinfo.cmsize;
cmap.red = &cmap_data[fbinfo.cmsize * 0];
cmap.green = &cmap_data[fbinfo.cmsize * 1];
cmap.blue = &cmap_data[fbinfo.cmsize * 2];
if(ioctl(tty, WSDISPLAYIO_GETCMAP, &cmap) == -1) {
perror("error getting color map");
close(tty), free(cmap_data);
return EXIT_FAILURE;
}
if(ioctl(tty, WSDISPLAYIO_PUTCMAP, &cmap) == -1) {
perror("error putting color map");
close(tty), free(cmap_data);
return EXIT_FAILURE;
}
free(cmap_data);
close(tty);
return EXIT_SUCCESS;
}
What am I doing wrong and how can I make it retrieve and restore color maps correctly?
Cause
I looked into the problem more and it appears that some kernel memory is either uninitialized or getting corrupted. Specifically, sc_cmap_red, sc_cmap_green, and sc_cmap_blue of struct vesafb_softc (in vesafbvar.h on lines 89 to 91) contain incorrect data. This is somewhat surprising, as lines 719 to 722 of vesafb.c initialize it:
/* Fill in the softc colourmap arrays */
sc->sc_cmap_red[i / 3] = rasops_cmap[i + 0];
sc->sc_cmap_green[i / 3] = rasops_cmap[i + 1];
sc->sc_cmap_blue[i / 3] = rasops_cmap[i + 2];
It contains incorrect data even if I move that out of the if statement it's in, so it might be getting corrupted rather than being uninitialized.
The driver is able to get and set color maps correctly, however; it just cannot seem to get the initial one in struct vesafb_softc right.
Workaround
A simple solution would be to make a program re-set the default color map. As the above snippet indicated, it was supposed to get its initial colors from rasops_cmap, which happens to be defined on lines 55 to 122 in rasops.c:
/* ANSI colormap (R,G,B). Upper 8 are high-intensity */
const u_char rasops_cmap[256*3] = {
/* ... */
};
With those colors, you can make a program that sets that as the current color map. I had to make a few changes so the cursor didn't disappear, but it mostly worked.
Better Solution
As I was looking around for more information, I had found this blog post. When I recompiled the kernel with genfb(4) rather than vesafb(4), the kernel hung at boot. It turns out this was because the bootloader I was using was not new enough to pass the required parameters to the kernel.
I happened to look at the NetBSD 6.0 changelog and noticed this entry:
amd64, i386
The bootloader has been enhanced to support framebuffer consoles using VESA BIOS extensions. These changes allow the x86 ports to work with the genfb(4) driver, and obsoletes the i386-only vesafb(4) driver. [jmcneill 20090216]
I downloaded NetBSD 6.0_BETA and booted it from the boot prompt like this:
> vesa 640x480x8
> boot netbsd
...and everything worked.
In short, using a newer version of NetBSD and ditching vesafb(4) solves the problem.

Jpeglib code gives garbled output, even the bundled example code?

I'm on Ubuntu Intrepid and I'm using jpeglib62 6b-14. I was working on some code, which only gave a black screen with some garbled output at the top when I tried to run it. After a few hours of debugging I got it down to pretty much the JPEG base, so I took the example code, wrote a little piece of code around it and the output was exactly the same.
I'm convinced jpeglib is used in a lot more places on this system and it's simply the version from the repositories so I'm hesitant to say that this is a bug in jpeglib or the Ubuntu packaging.
I put the example code below (most comments stripped). The input JPEG file is an uncompressed 640x480 file with 3 channels, so it should be 921600 bytes (and it is). The output image is JFIF and around 9000 bytes.
If you could help me with even a hint, I'd be very grateful.
Thanks!
#include <stdio.h>
#include <stdlib.h>
#include "jpeglib.h"
#include <setjmp.h>
int main ()
{
// read data
FILE *input = fopen("input.jpg", "rb");
JSAMPLE *image_buffer = (JSAMPLE*) malloc(sizeof(JSAMPLE) * 640 * 480 * 3);
if(input == NULL or image_buffer == NULL)
exit(1);
fread(image_buffer, 640 * 3, 480, input);
// initialise jpeg library
struct jpeg_compress_struct cinfo;
struct jpeg_error_mgr jerr;
cinfo.err = jpeg_std_error(&jerr);
jpeg_create_compress(&cinfo);
// write to foo.jpg
FILE *outfile = fopen("foo.jpg", "wb");
if (outfile == NULL)
exit(1);
jpeg_stdio_dest(&cinfo, outfile);
// setup library
cinfo.image_width = 640;
cinfo.image_height = 480;
cinfo.input_components = 3; // 3 components (R, G, B)
cinfo.in_color_space = JCS_RGB; // RGB
jpeg_set_defaults(&cinfo); // set defaults
// start compressing
int row_stride = 640 * 3; // number of characters in a row
JSAMPROW row_pointer[1]; // pointer to the current row data
jpeg_start_compress(&cinfo, TRUE); // start compressing to jpeg
while (cinfo.next_scanline < cinfo.image_height) {
row_pointer[0] = & image_buffer[cinfo.next_scanline * row_stride];
(void) jpeg_write_scanlines(&cinfo, row_pointer, 1);
}
jpeg_finish_compress(&cinfo);
// clean up
fclose(outfile);
jpeg_destroy_compress(&cinfo);
}
You're reading a JPEG file into memory (without decompressing it) and writing out that buffer as if it were uncompressed, that's why you're getting garbage. You need to decompress the image first before you can feed it into the JPEG compressor.
In other words, the JPEG compressor assumes that its input is raw pixels.
You can convert your input image into raw RGB using ImageMagick:
convert input.jpg rgb:input.raw
It should be exactly 921600 bytes in size.
EDIT: Your question is misleading when you state that your input JPEG file in uncompressed. Anyway, I compiled your code and it works fine, compresses the image correctly. If you can upload the file you're using as input, it might be possible to debug further. If not, I suggest you test your program using an image created from a known JPEG using ImageMagick:
convert some_image_that_is_really_a_jpg.jpg -resize 640x480! rgb:input.jpg
You are reading the input file into memmory compressed and then you are recompressing it before righting to file. You need to decompress the image_buffer before compressing it again. Or alternativly instead of reading in a jpeg read a .raw image
What exactly do you mean by "The input JPEG file is an uncompressed"? Jpegs are all compressed.
In your code, it seems that in the loop you give one row of pixels to libjpeg and ask it to compress it. It doesn't work that way. libjpeg has to have at least 8 rows to start compression (sometimes even more, depending on parameters). So it's best to leave libjpeg to control the input buffer and don't do its job for it.
I suggest you read how cjpeg.c does its job. The easiest way I think is to put your data in a raw type known by libjpeg (say, BMP), and use libjpeg to read the BMP image into its internal representation and compress from there.

Resources