I want to pack some compressed video packets(h.264) to ".mp4" container.
One word, Muxing, no decoding and no encoding.
And I have no idea how to set pts, dts and duration.
I get the packets with "pcap" library.
I removed headers before compressed video data show up. e.g. Ethernet, VLAN.
I collected data until one frame and decoded it for getting information of data. e.g. width, height. (I am not sure that it is necessary)
I initialized output context, stream and codec context.
I started to receive packets with "pcap" library again. (now for muxing)
I made one frame and put that data in AVPacket structure.
I try to set PTS, DTS and duration. (I think here is wrong part, not sure though)
*7-1. At the first frame, I saved time(msec) with packet header structure.
*7-2. whenever I made one frame, I set parameters like this : PTS(current time - start time), DTS(same PTS value), duration(current PTS - before PTS)
I think it has some error because :
I don't know how far is suitable long for dts from pts.
At least, I think duration means how long time show this frame from now to next frame, so It should have value(next PTS - current PTS), but I can not know the value next PTS at that time.
It has I-frame only.
// make input context for decoding
AVFormatContext *&ic = gInputContext;
ic = avformat_alloc_context();
AVCodec *cd = avcodec_find_decoder(AV_CODEC_ID_H264);
AVStream *st = avformat_new_stream(ic, cd);
AVCodecContext *cc = st->codec;
avcodec_open2(cc, cd, NULL);
// make packet and decode it after collect packets is be one frame
gPacket.stream_index = 0;
gPacket.size = gPacketLength[0];
gPacket.data = gPacketData[0];
gPacket.pts = AV_NOPTS_VALUE;
gPacket.dts = AV_NOPTS_VALUE;
gPacket.flags = AV_PKT_FLAG_KEY;
avcodec_decode_video2(cc, gFrame, &got_picture, &gPacket);
// I checked automatically it initialized after "avcodec_decode_video2"
// put some info that I know that not initialized
cc->time_base.den = 90000;
cc->time_base.num = 1;
cc->bit_rate = 2500000;
cc->gop_size = 1;
// make output context with input context
AVFormatContext *&oc = gOutputContext;
avformat_alloc_output_context2(&oc, NULL, NULL, filename);
AVFormatContext *&ic = gInputContext;
AVStream *ist = ic->streams[0];
AVCodecContext *&icc = ist->codec;
AVStream *ost = avformat_new_stream(oc, icc->codec);
AVCodecContext *occ = ost->codec;
avcodec_copy_context(occ, icc);
occ->flags |= CODEC_FLAG_GLOBAL_HEADER;
avio_open(&(oc->pb), filename, AVIO_FLAG_WRITE);
// repeated part for muxing
AVRational Millisecond = { 1, 1000 };
gPacket.stream_index = 0;
gPacket.data = gPacketData[0];
gPacket.size = gPacketLength[0];
gPacket.pts = av_rescale_rnd(pkthdr->ts.tv_sec * 1000 /
+ pkthdr->ts.tv_usec / 1000 /
- gStartTime, Millisecond.den, ost->time_base.den, /
(AVRounding)(AV_ROUND_NEAR_INF | AV_ROUND_PASS_MINMAX));
gPacket.dts = gPacket.pts;
gPacket.duration = gPacket.pts - gPrev;
gPacket.flags = AV_PKT_FLAG_KEY;
gPrev = gPacket.pts;
av_interleaved_write_frame(gOutputContext, &gPacket);
Expected and actual results is a .mp4 video file that can play.
Related
I want to demux and then mux .avi file without changing anything.
My program is this (redacted for brevity):
AVFormatContext *input_format_context = NULL;
avformat_open_input(
&input_format_context,
input_url,
NULL, // fmt
NULL // options
);
avformat_find_stream_info(input_format_context, NULL);
AVFormatContext *output_format_context = NULL;
avformat_alloc_output_context2(
&output_format_context,
NULL, // oformat
NULL, // format_name
output_url
);
avio_open2(
&output_format_context->pb,
output_url,
AVIO_FLAG_WRITE,
NULL, // int_cb,
NULL // options
);
for (int i = 0; i < input_format_context->nb_streams; i++) {
avformat_new_stream(output_format_context, NULL);
AVStream *input_stream = input_format_context->streams[i];
AVStream *output_stream = output_format_context->streams[i];
AVCodecParameters *params = avcodec_parameters_alloc();
avcodec_parameters_copy(params, input_stream->codecpar);
output_stream->codecpar = params;
}
avformat_write_header(output_format_context, NULL))
AVPacket *input_packet = NULL;
input_packet = av_packet_alloc();
while (!av_read_frame(
input_format_context,
input_packet
)) {
av_write_frame(output_format_context, input_packet);
av_packet_unref(input_packet);
}
av_write_trailer(output_format_context);
Problem:
Output file is created but instead of close to 10 minute video it is a 24-second slide show consisting of around 3 frames.
It seems that the problem is (perhaps not the only one) lack of PTS on the packet.
When I explicitly print it (input_packet->pts) for each packet it is -9223372036854775808. And also the following warning is printed:
[avi # 0x562868c6c000] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
How do I then "fix my code to set the timestamps properly"?
I just found a solution.
I added this:
output_stream->time_base = input_stream->time_base;
which then, I understand, allows the video player to calculate PTS on the fly.
This does not remove the warning itself, though. I understand that .avi simply does not have PTS, so it's not a bug as such. To get rid of the warning one can manually set PTS on the packets:
input_packet->pts = calculated_ts;
I would think I should be able to also just do:
output_format_context->oformat->flags |= AVFMT_NOTIMESTAMPS;
However, I cannot do that:
error: assignment of member ‘flags’ in read-only object
So, it looks like ffmpeg is requiring PTS even for .avi or there's a bug or I'm still doing something wrong.
I'm writing program with libav/ffmpeg to download internet radio stream and play it on soundcard with alsa.
I've managed to download stream and extract packet and frame.
I'm having problem with av_write_header() function which (according to this https://www.ffmpeg.org/doxygen/3.2/group__lavf__encoding.html#details) I must call. It crashes and gives me the following error:
[alsa # 0x55d7ba32e580] sample format 0x15001 is not supported
Number 0x15001 is 86017 in decimal, which is index in enum AVCodecID of MP3 format(AV_CODEC_ID_MP3) used by this stream. The sample format has index 3. I can't figure out why libav parses the header wrong.
Here is a part of my code that is responsible for configuring output:
avdevice_register_all();
AVOutputFormat *output = av_guess_format("alsa",NULL,NULL);
AVFormatContext *outputFormatContext = avformat_alloc_context();
outputFormatContext->oformat = output;
outputFormatContext->flags = AVFMT_NOFILE;
AVStream *stream = avformat_new_stream(outputFormatContext,NULL);
AVCodecParameters *oCodecParameters = avcodec_parameters_alloc();
ret = avcodec_parameters_copy(oCodecParameters,iCodecParameters);
if(ret < 0){
printf("avformat_parameters_copy\n");
exit(0);
}
stream->codecpar = oCodecParameters;
if(avformat_write_header(outputFormatContext,NULL)<0){
dumpParameters(stream->codecpar);
printf("avformat_write_header\n");
exit(0);
}
The full code is here: https://github.com/szymonbarszcz99/C-internet-radio
It seems that in libav we can't do simple copy. Instead I have to manually give it requested parameters. Changing avcodec_parameters_copy() to this
AVCodecParameters *oCodecParameters = avcodec_parameters_alloc();
oCodecParameters->format = 8;
oCodecParameters->codec_type = 1;
oCodecParameters->sample_rate = 44100;
oCodecParameters->channels = 2;
stream->codecpar = oCodecParameters;
fixes this problem
I was working on mp4 file creation project using FFMPEG, i tried to convert the stream information of video packet based on FFMPEG muxing,remuxing code, but header get damaged after convert into so file is corrupted.
/* this code used to set the stream information */
AVFormatContext *input_context,*output_context;
AVDictionary *opt;
AVStream *out_stream;
AVCodecContext *newcontext = NULL;
out_stream= avformat_new_stream(output_context,NULL);
newcontext = avcodec_alloc_context3(codec);
newcontext->codec_id=Output_fmt->video_codec;
newcontext->bit_rate =in_stream->codec->bit_rate;
newcontext->width = in_stream->codec->width;
newcontext->height = in_stream->codec->height;
newcontext->timecode_frame_start = in_stream->codec->timecode_frame_start;
newcontext->gop_size = in_stream->codec->gop_size;
newcontext->profile = in_stream->codec->profile;
newcontext->level = in_stream->codec->level;
newcontext->pix_fmt = PIX_FMT_YUV420P;
newcontext->frame_size = in_stream->codec->frame_size;
newcontext->sample_fmt = in_stream->codec->sample_fmt;
newcontext->sample_rate = in_stream->codec->sample_rate;
time_base = (double)in_stream->time_base.num / (double)in_stream->time_base.den;
duration = (double)in_stream->duration * time_base * 1000.0;
if (!out_stream) {
fprintf(stderr, "Failed allocating output stream\n");
ret = AVERROR_UNKNOWN;
return;
}
ret = avcodec_copy_context(out_stream->codec,newcontext);
if (ret < 0) {
fprintf(stderr, "Failed to copy context from input to output stream codec context\n");
goto end;
}
out_stream->codec->codec_tag = 0;
if (output_context->oformat->flags & AVFMT_GLOBALHEADER)
out_stream->codec->flags |= AV_CODEC_FLAG_GLOBAL_HEADER;
Changed the Header Information using:
/* this code used to set the metadata */
av_dict_set(&opt, "major_brand", "mp42", 0);
av_dict_set(&opt, "minor_version","512" , 0);
av_dict_set(&opt, "compatible_brands","isomiso2avc1mp41",0);
av_dict_set(&opt, "comment","Hash=855738390",0);
output_context->metadata = opt;
ret = avformat_write_header(output_context,NULL);
after create the mp4 file check file using ffmpeg in terminal. getting Error like this:
/this error message/
[mpeg4 # 0x7ff2b9811c00] header damaged Last message repeated 39
times [mov,mp4,m4a,3gp,3g2,mj2 # 0x7ff2ba800000] decoding for stream 0
failed [mov,mp4,m4a,3gp,3g2,mj2 # 0x7ff2ba800000] Could not find codec
parameters for stream 0 (Video: mpeg4 (mp4v / 0x7634706D), none, 376
kb/s): unspecified size Consider increasing the value for the
'analyzeduration' and 'probesize' options.
Easiest thing is to just download a freeware hex editor (for your specific O.S). Next is use desktop (commandline) version of FFmpeg (download a static build)
Use the commandline FFmpeg to convert Source to MP4 (ie: as mp4_ffmpeg.mp4)
Use your code to convert Source to MP4 (ie: as mp4_code.mp4)
Open both mp4_ffmpeg.mp4 & mp4_code.mp4 and compare bytes. The working one should be mp4_ffmpeg.mp4 so what's different from bytes produced with your code?
Things to look for :
All begin with ftyp?
moov is header and should be at start (sometimes at back after mdat which holds all a/v data in one chunk. To move the header of any mp4 to front or beginning bytes then use -movflags +faststart for example in commandline use : ffmpeg -i myfile.avi -movflags +faststart newfile.mp4)
Before each of the words moov or mdat, the previous 4 bytes are the size (in bytes) after you skip the 4 letters of word... are these sizes correct?
Do you have all the MP4 atoms (metadata sections) defined? They match what FFmpeg produced for its version of the MP4 converting?
I want my app (which works with RGBA8888 images) to be able to paste images from the Windows clipboard. So it should be able to read images off the clipboard that come from any common raster image apps like Gimp, Photoshop, MSPaint, etc.
From reading up on the clipboard functions, it seems I should be able to call GetClipboardData(CF_DIBV5) to get access to pretty much any bitmap type that's on the Clipboard since Windows automatically converts between that and CF_BITMAP and CF_DIB. But from reading up on the DIB format, I see that there is an immense number of possible combinations of bit depth, RGB order, optional compression, etc. It seems like what I'm doing would be a common task, but I don't see any conversion functions in the Windows API (unless I'm poor at searching), and this seems like something that would take a week to write to support all possible formats. So I'm wondering if I've overlooked something obvious. Or if there is some kind of assumption I can make to simplify this...like if all the popular image apps happen to copy images to the clipboard in uncompressed/unindexed formats.
UPDATE: Here's what I have so far:
HGLOBAL clipboard = GetClipboardData(CF_DIBV5);
exists = clipboard != NULL;
int dataLength = GlobalSize(clipboard);
exists = dataLength != 0;
if (exists) {
LPTSTR lockedClipboard = GlobalLock(clipboard);
exists = lockedClipboard != NULL;
if (exists) {
BITMAPV5HEADER *header = (BITMAPV5HEADER*)lockedClipboard;
LONG width = header->bV5Width;
LONG height = header->bV5Height;
BYTE *bits = header + sizeof(header) + header->bV5ClrUsed * sizeof(RGBQUAD);
//Now what? Need function to convert the bits to something uncompressed.
GlobalUnlock(clipboard);
}
}
UPDATE 2:
To clarify, I need literally uncompressed 32 bit image data (RRGGBBAA) which I can manipulate however I like in a cross-platform app. I have no need to use Windows APIs to draw this image to screen.
I am aware of a 3rd party library called stdb_image.h that can load .bmps, .jpgs, and .pngs into the type of data I need. So if there's a way I can turn the clipboard data into bitmap or png file data without losing alpha, then I'll be in good shape.
The basic strategy I've found is to check if there's a raw PNG on the clipboard and use that first if available. That's the easiest. Some apps, such as GIMP, copy images as PNG to the clipboard.
Then check for CF_DIBV5. The location of the actual bits depends on whether the "compression" is BI_BITFIELDS:
int offset = bitmapV5Header->bV5Size + bitmapV5Header->bV5ClrUsed * (bitmapV5Header->bV5BitCount > 24 ? sizeof(RGBQUAD) : sizeof(RGBTRIPLE));
if (compression == BI_BITFIELDS)
offset += 12; //bit masks follow the header
BYTE *bits = (BYTE*)bitmapV5Header + offset;
If the header says compression is BI_BITFIELDS, then the data is already as I needed it.
If the header says compression is BI_RGB and the bit count is 24 or 32, then I can unpack the bytes. 24 bytes means row size might not land on a DWORD boundary, so you have to watch for that.
Finally, lower bit counts than 24 likely mean indexed color, which I don't have working yet.
Here is example of usage for CF_DIBV5 and CF_DIB. It's best to use CF_DIB as backup option. Note, this code won't work for palette based images (if it is not guaranteed 32bit then see the method further down)
You can use SetDIBitsToDevice to draw directly on HDC, or use SetDIBits
GDI functions don't support alpha transparency (except for a couple of functions like TransparentBlt), in general you have to use libraries such as GDI+ for that.
void foo(HDC hdc)
{
if (!OpenClipboard(NULL))
return;
HANDLE handle = GetClipboardData(CF_DIBV5);
if (handle)
{
BITMAPV5HEADER* header = (BITMAPV5HEADER*)GlobalLock(handle);
if (header)
{
BITMAPINFO bmpinfo;
memcpy(&bmpinfo.bmiHeader, header, sizeof(BITMAPINFOHEADER));
bmpinfo.bmiHeader.biSize = sizeof(BITMAPINFO);
//(use `header` to access other BITMAPV5HEADER information)
int w = bmpinfo.bmiHeader.biWidth;
int h = bmpinfo.bmiHeader.biHeight;
const char* bits = (char*)(header) + header->bV5Size;
//draw using SetDIBitsToDevice
SetDIBitsToDevice(hdc,0,0,w,h,0,0,0,h,bits,&bmpinfo,DIB_RGB_COLORS);
}
}
else
{
handle = GetClipboardData(CF_DIB);
if (handle)
{
BITMAPINFO* bmpinfo = (BITMAPINFO*)GlobalLock(handle);
if (bmpinfo)
{
int w = bmpinfo->bmiHeader.biWidth;
int h = bmpinfo->bmiHeader.biHeight;
const char* bits = (char*)(bmpinfo)+bmpinfo->bmiHeader.biSize;
SetDIBitsToDevice(hdc, 0, 0, w, h, 0, 0, 0, h, bits, bmpinfo, 0);
}
}
}
CloseClipboard();
}
If the original image is palette based, you would have to convert to 32bit. Alternatively you could add BITMAPFILEHEADER to the data (assuming the source is bitmap) then pass to the other library.
This is an example using CreateDIBitmap and GetDIBits to make sure the pixels are in 32bit:
HANDLE handle = GetClipboardData(CF_DIB);
if (handle)
{
BITMAPINFO* bmpinfo = (BITMAPINFO*)GlobalLock(handle);
if (bmpinfo)
{
int offset = (bmpinfo->bmiHeader.biBitCount > 8) ?
0 : sizeof(RGBQUAD) * (1 << bmpinfo->bmiHeader.biBitCount);
const char* bits = (const char*)(bmpinfo)+bmpinfo->bmiHeader.biSize + offset;
HBITMAP hbitmap = CreateDIBitmap(hdc, &bmpinfo->bmiHeader, CBM_INIT, bits, bmpinfo, DIB_RGB_COLORS);
//convert to 32 bits format (if it's not already 32bit)
BITMAP bm;
GetObject(hbitmap, sizeof(bm), &bm);
int w = bm.bmWidth;
int h = bm.bmHeight;
char *bits32 = new char[w*h*4];
BITMAPINFOHEADER bmpInfoHeader = { sizeof(BITMAPINFOHEADER), w, h, 1, 32 };
HDC hdc = GetDC(0);
GetDIBits(hdc, hbitmap, 0, h, bits32, (BITMAPINFO*)&bmpInfoHeader, DIB_RGB_COLORS);
ReleaseDC(0, hdc);
//use bits32 for whatever purpose...
//cleanup
delete[]bits32;
}
}
I am trying to use the C interface of CoreGraphics & CoreFoundation to save a buffer of 32-bit RGBA data (as a void*) to a PNG file. When I try to finialize the CGImageDestinationRef, the following error message is printed to the console:
libpng error: No IDATs written into file
As far as I can tell, the CGImageRef I'm adding to the CGImageDestinationRef is valid.
Relavent Code:
void saveImage(const char* szImage, void* data, size_t dataSize, size_t width, size_t height)
{
CFStringRef name = CFStringCreateWithCString(NULL, szImage, kCFStringEncodingASCII);
CFURLRef texture_url = CFURLCreateWithFileSystemPath(
NULL,
name,
kCFURLPOSIXPathStyle,
false);
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
CGDataProviderRef dataProvider = CGDataProviderCreateWithData(NULL, data, dataSize, NULL);
CGImageRef image = CGImageCreate(width, height, 8, 32, 32 * width, colorSpace,
kCGImageAlphaLast | kCGBitmapByteOrderDefault, dataProvider,
NULL, FALSE, kCGRenderingIntentDefault);
// From Image I/O Programming Guide, "Working with Image Destinations"
float compression = 1.0; // Lossless compression if available.
int orientation = 4; // Origin is at bottom, left.
CFStringRef myKeys[3];
CFTypeRef myValues[3];
CFDictionaryRef myOptions = NULL;
myKeys[0] = kCGImagePropertyOrientation;
myValues[0] = CFNumberCreate(NULL, kCFNumberIntType, &orientation);
myKeys[1] = kCGImagePropertyHasAlpha;
myValues[1] = kCFBooleanTrue;
myKeys[2] = kCGImageDestinationLossyCompressionQuality;
myValues[2] = CFNumberCreate(NULL, kCFNumberFloatType, &compression);
myOptions = CFDictionaryCreate( NULL, (const void **)myKeys, (const void **)myValues, 3,
&kCFTypeDictionaryKeyCallBacks, &kCFTypeDictionaryValueCallBacks);
CFStringRef type = CFStringCreateWithCString(NULL, "public.png", kCFStringEncodingASCII);
CGImageDestinationRef dest = CGImageDestinationCreateWithURL(texture_url, type, 1, myOptions);
CGImageDestinationAddImage(dest, image, NULL);
if (!CGImageDestinationFinalize(dest))
{
// ERROR!
}
CFRelease(image);
CFRelease(colorSpace);
CFRelease(dataProvider);
CFRelease(dest);
CFRelease(texture_url);
}
This post is similar, except I'm not using the Objective C interface: Saving a 32 bit RGBA buffer into a .png file (Cocoa OSX)
Answering my own questions:
In addition to the issues pointed out by NSGod, the IDAT issue was an invalid parameter to CGImageCreate(): parameter 5 is bytesPerRow, not bitsPerRow. So 32 * width was incorrect; 4 * width is correct.
Despite what this page of the official documentation lists, UTCoreTypes.h is located in the CoreServices.framework for MacOSX, not MobileCoreServices.framework.
There are numerous issues with your code.
Here it is rewritten how I would do it:
void saveImage(const char* szImage, void* data, size_t dataSize, size_t width, size_t height)
{
CFStringRef name = CFStringCreateWithCString(NULL, szImage, kCFStringEncodingUTF8);
CFURLRef texture_url = CFURLCreateWithFileSystemPath(
NULL,
name,
kCFURLPOSIXPathStyle,
false);
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
CGDataProviderRef dataProvider = CGDataProviderCreateWithData(NULL, data,
dataSize, NULL);
CGImageRef image = CGImageCreate(width, height, 8, 32, 32 * width, colorSpace,
kCGImageAlphaLast | kCGBitmapByteOrderDefault,
dataProvider, NULL, FALSE, kCGRenderingIntentDefault);
// From Image I/O Programming Guide, "Working with Image Destinations"
float compression = 1.0; // Lossless compression if available.
int orientation = 4; // Origin is at bottom, left.
CFStringRef myKeys[3];
CFTypeRef myValues[3];
CFDictionaryRef myOptions = NULL;
myKeys[0] = kCGImagePropertyOrientation;
myValues[0] = CFNumberCreate(NULL, kCFNumberIntType, &orientation);
myKeys[1] = kCGImagePropertyHasAlpha;
myValues[1] = kCFBooleanTrue;
myKeys[2] = kCGImageDestinationLossyCompressionQuality;
myValues[2] = CFNumberCreate(NULL, kCFNumberFloatType, &compression);
myOptions = CFDictionaryCreate(NULL, (const void **)myKeys,
(const void **)myValues, 3, &kCFTypeDictionaryKeyCallBacks,
&kCFTypeDictionaryValueCallBacks);
CGImageDestinationRef dest =
CGImageDestinationCreateWithURL(texture_url, kUTTypePNG, 1, NULL);
CGImageDestinationAddImage(dest, image, NULL);
CGImageDestinationSetProperties(dest, myOptions);
if (!CGImageDestinationFinalize(dest))
{
// ERROR!
}
}
First, never use ASCII when dealing with file system paths, use UTF8. Second, you were constructing a dictionary to be used to set the properties of the image, but you were using it with the wrong function. The documentation for CGImageDestinationCreateWithURL() says the following:
CGImageDestinationCreateWithURL
Creates an image destination that writes to a location specified by a
URL.
CGImageDestinationRef CGImageDestinationCreateWithURL (
CFURLRef url,
CFStringRef type,
size_t count,
CFDictionaryRef options
);
Parameters
options - Reserved for future use. Pass NULL.
You were trying to pass a dictionary of properties when you were supposed to pass NULL. (Also, you can simply use the kUTTypePNG Uniform Type Identifier string constant instead of re-creating it). First call CGImageDestinationCreateWithURL(), then call CGImageDestinationAddImage() to add the image, then call CGImageDestinationSetProperties() and pass in the dictionary of properties you created.
[UPDATE]: If after these changes you're still having libpng error: No IDATs written into file issues, try the following: First, make sure that dataProvider is non-NULL-- in other words, make sure the CGDataProviderCreateWithData() function succeeded. Second, if dataProvider is valid, perhaps try changing the options from kCGImageAlphaLast | kCGBitmapByteOrderDefault to simply kCGImageAlphaPremultipliedLast and see if it succeeds.