I have been trying to output video (from my webcam) simultaneously to both a file ('out.mkv') and pipe:
The file gets filtered frames, and the pipe: gets unfiltered rawvideo.
My frame rate is 30 fps. However, I am getting a far lower framerate in my file output.
Attached is the while loop which reads packets and writes them to output:
while (1) {
av_read_frame(ifmt_ctx, packet);
stream_index = packet->stream_index;
StreamContext *stream = &file_stream_ctx[stream_index];
av_packet_rescale_ts(packet,
ifmt_ctx->streams[stream_index]->time_base,
stream->dec_ctx->time_base);
avcodec_send_packet(stream->dec_ctx, packet);
while (ret >= 0) {
avcodec_receive_frame(stream->dec_ctx, stream->dec_frame);
stream->dec_frame->pts = stream->dec_frame->best_effort_timestamp;
ret = filter_encode_write_frame(stream->dec_frame, stream_index,file_stream_ctx,file_filter_ctx, file_ofmt_ctx);
ret = av_interleaved_write_frame(pipe_ofmt_ctx, packet);
}
}
'ifmt_ctx' is the AVFormatContext for the webcam.
'file_ofmt_ctx', is the AVFormatContext for the pipe for the output file, and pipe_ofmt_ctx is the AVFormatContext.
'file_stream_ctx' and 'file_filter_ctx' are the stream and filter contexts used for filtering and encoding the file output.
My guess is that writing to the pipe is taking too long and not allowing the next packet to be read on time - causing a lower frame rate. Does that make sense? If so - any suggestions on how to fix it? (I tried using AVFMT_FLAG_NONBLOCK but it doesn't seem to help).
Thanks
Hillel
It is hard to tell without any profiling, but you're probably correct given what you're seeing. I'd be curious to know if you eliminate one of the write outs if you see better performance.
I always write the packets or frames (depending on the case) into a queue and have a separate thread do writing. You might even want two threads - one for the video (packet queue) and one for the frames.
As an aside, I think the line avcodec_send_packet(stream->dec_ctx, packet); consumes the packet, so when you use it later in ret = av_interleaved_write_frame(pipe_ofmt_ctx, packet); that seems like it wouldn't work. You could try copying the packet and see if that helps at all:
av_packet_ref(new_packet, packet);
Assuming you've allocated new_packet of course. Doesn't sound like what your seeing is this, so it might be OK what you're doing. But something you could also try.
Related
I'm using libavformat to read packets from rtsp and remux it to mp4 (fragmented).
Video frames are intact, meaning I don't want to transcode/modify/change anything.
Video frames shall be remuxed into mp4 in their original form. (i.e.: NALUs shall remain the same).
I have updated libavformat to latest (currently 4.4).
Here is my snippet:
//open input, probesize is set to 32, we don't need to decode anything
avformat_open_input
//open output with custom io
avformat_alloc_output_context2(&ofctx,...);
ofctx->pb = avio_alloc_context(buffer, bufsize, 1/*write flag*/, 0, 0, &writeOutput, 0);
ofctx->flags |= AVFMT_FLAG_NOBUFFER | AVFMT_FLAG_FLUSH_PACKETS | AVFMT_FLAG_CUSTOM_IO;
avformat_write_header(...);
//loop
av_read_frame()
LOGPACKET_DETAILS //<- this works, packets are coming
av_write_frame() //<- this doesn't work, my write callback is not called. Also tried with av_write_interleaved_frame, not seem to work.
int writeOutput(void *opaque, uint8_t *buffer, int buffer_size) {
printf("writeOutput: writing %d bytes: ", buffer_size);
}
avformat_write_header works, it prints the header correctly.
I'm looking for the reason on why my custom IO is not called after a frame has been read.
There must be some more flags should be set to ask avformat to don't care about decoding, just write out whatever comes in.
More information:
Input stream is a VBR encoded H264. It seems av_write_frame calls my write function only in case an SPS, PPS or IDR frame. Non-IDR frames are not passed at all.
Update
I found out if I request IDR frame at every second (I can ask it from the encoder), writeOutput is called at every second.
I created a test: after a client joins, I requested the encoder to create IDRs #1Hz for 10 times. Libav calls writeOutput at 1Hz for 10 seconds, but then encoder sets itself back to create IDR only at every 10 seconds. And then libav calls writeOutput only at every 10s, which makes my decoder fail. In case 1Hz IDRs, decoder is fine.
This must be a stupid question because this should be a very common and simple problem, but I haven't been able to find an answer anywhere, so I'll bite the bullet and ask.
How on earth should I go about reading from the standard input when there is no way of determining the size of the data? Obviously if the data ends in some kind of terminator like a NUL or EOF then this is quite trivial, but my data does not. This is simple IPC: the two programs need to talk back and forth and ending the file streams with EOF would break everything.
I thought this should be fairly simple. Clearly programs talk to each other over pipes all the time without needing any arcane tricks, so I hope there is a simple answer that I'm too stupid to have thought of. Nothing I've tried has worked.
Something obvious like (ignoring necessary realloc's for brevity):
int size = 0, max = 8192;
unsigned char *buf = malloc(max);
while (fread((buf + size), 1, 1, stdin) == 1)
++size;
won't work since fread() blocks and waits for data, so this loop won't terminate. As far as I know nothing in stdio allows nonblocking input, so I didn't even try any such function. Something like this is the best I could come up with:
struct mydata {
unsigned char *data;
int slen; /* size of data */
int mlen; /* maximum allocated size */
};
...
struct mydata *buf = xmalloc(sizeof *buf);
buf->data = xmalloc((buf->mlen = 8192));
buf->slen = 0;
int nread = read(0, buf->data, 1);
if (nread == (-1))
err(1, "read error");
buf->slen += nread;
fcntl(0, F_SETFL, oflags | O_NONBLOCK);
do {
if (buf->slen >= (buf->mlen - 32))
buf->data = xrealloc(buf->data, (buf->mlen *= 2));
nread = read(0, (buf->data + buf->slen), 1);
if (nread > 0)
buf->slen += nread;
} while (nread == 1);
fcntl(0, F_SETFL, oflags);
where oflags is a global variable containing the original flags for stdin (cached at the start of the program, just in case). This dumb way of doing it works as long as all of the data is present immediately, but fails otherwise. Because this sets read() to be non-blocking, it just returns -1 if there is no data. The program communicating with mine generally sends responses whenever it feels like it, and not all at once, so if the data is at all large this exits too early and fails.
How on earth should I go about reading from the standard input when there is no way of determining the size of the data?
There always has to be a way to determinate the size. Otherwise, the program would require infinite memory, and thus impossible to run on a physical computer.
Think about it this way: even in the case of a never-ending stream of data, there must be some chunks or points where you have to process it. For instance, a live-streamed video has to decode a portion of it (e.g. a frame). Or a video game which processes messages one by one, even if the game has undetermined length.
This holds true regardless of the type of I/O you decide to use (blocking/non-blocking, synchronous/asynchronous...). For instance, if you want to use typical blocking synchronous I/O, what you have to do is process the data in a loop: each iteration, you read as much data as is available, and process as much as you can. Whatever you can not process (because you have not received enough yet), you keep for the next iteration. Then, the rest of the loop is the rest of the logic of the program.
In the end, regardless of what you do, you (or someone else, e.g. a library, the operating system, the hardware buffers...) have to buffer incoming data until it can be processed.
Basically, you have two choices -- synchronous or asynchronous -- and both have their advantages and disadvantages.
For synchronous, you need either delimeters or a length field embedded in the record (or fixed length records, but that is pretty inflexible). This works best for synchronous protocols like synchronous rpc or simplex client-server interactions where only one side talks at a time while the other side waits. For ASCII/text based protocols, it is common to use a control-character delimiter like NL/EOL or NUL or CTX to mark the end of messages. Binary protocols more commonly use an embedded length field -- the receiver first reads the length and then reads the full amount of (expected) data.
For asynchronous, you use non-blocking mode. It IS possible to use non-blocking mode with stdio streams, it just requires some care. out-of-data conditions show up to stdio like error conditions, so you need to use ferror and clearerr on the FILE * as appropriate.
It's possible for both to be used -- for example in client-server interactions, the clients may use synchronous (they send a request and wait for a reply) while the server uses asynchronous (to be be robust in the presence of misbehaving clients).
The read api on Linux or the ReadFile Api on windows will immediately return and not wait for the specified number of bytes to fill the buffer (when reading a pipe or socket). Read then reurns the number of bytes read.
This means, when reading from a pipe, you set a buffersize, read as much as returned and the process it. You then read the next bit. The only time you are blocked is if there is no data available at all.
This differs from fread which only returns once the desired number of bytes are returned or the stream determines doing so is impossible (like eof).
I'm trying to generate short tones of variable lengths through ALSA in a small C program. A few examples I've tried work fine when playing one second worth of sound but anything shorter than that just doesn't produce any sound at all.
I'm filling a buffer with a sine wave like so:
#define BUFFER_LEN 44100
int freq = 700; //audio frequency
int fs = 44100; //sampling frequency
float buffer [BUFFER_LEN];
for (k=0; k<BUFFER_LEN; k++) {
buffer[k] = (sin(2*M_PI*freq/fs*k));
}
Setting the pcm device parameters:
if ((err = snd_pcm_set_params(handle,
SND_PCM_FORMAT_FLOAT,
SND_PCM_ACCESS_RW_INTERLEAVED,
1,
44100,
1,
500000)) < 0) {
printf("Playback open error: %s\n", snd_strerror(err));
exit(EXIT_FAILURE);
}
Playback:
frames = snd_pcm_writei(handle, buffer, BUFFER_LEN);
But if I want to change it to play the tone, say a quarter of a second (i.e. changing BUFFER_LEN to 11025) nothing comes out of the speaker anymore.
I've tried changing types from floats to shorts, setting the PCM_FORMAT to other values and trying different ways of filling the buffer with sine waves.
If anything, I hear a little 'bip' sound in the speakers but just not what I expect. The program doesn't segfaults or crash but I'm just puzzled how to make ALSA play shorter samples.
I don't know if I need to work with some exact frame or buffer size multiple but I'm very open to suggestions.
Your application writing samples to the buffer, and the hardware reading samples from the buffer and playing them are two different processes that run asynchronously.
If there is not enough free space in the buffer for the amount of samples you're trying to write, then snd_pcm_writei() will wait until enough space is available. But when snd_pcm_writei() returns, up to a full buffer of samples might not yet have been played.
To wait until the buffer is empty, use snd_pcm_drain().
I'm very new to C++, but I'm trying to learn some basics of TCP socket coding. Anyway, I've been able to send and receive messages, but I want to prefix my packets with the length of the packet (like I did in C# apps I made in the past) so when my window gets the FD_READ command, I have the following code to read just the first two bytes of the packet to use as a short int.
char lengthBuffer[2];
int rec = recv(sck, lengthBuffer, sizeof(lengthBuffer), 0);
short unsigned int toRec = lengthBuffer[1] << 8 | lengthBuffer[0];
What's confusing me is that after a packet comes in the 'rec' variable, which says how many bytes were read is one, not two, and if I make the lengthBuffer three chars instead of two, it reads three bytes, but if it's four, it also reads three (only odd numbers). I can't tell if I'm making some really stupid mistake here, or fundamentally misunderstanding some part of the language or the API. I'm aware that recv doesn't guarantee any number of bytes will be read, but if it's just two, it shouldn't take multiple reads.
Because you cannot assume how much data will be available, you'll need to continuously read from the socket until you have the amount you want. Something like this should work:
ssize_t rec = 0;
do {
int result = recv(sck, &lengthBuffer[rec], sizeof(lengthBuffer) - rec, 0);
if (result == -1) {
// Handle error ...
break;
}
else if (result == 0) {
// Handle disconnect ...
break;
}
else {
rec += result;
}
}
while (rec < sizeof(lengthBuffer));
Streamed sockets:
The sockets are generally used in a streamed way: you'll receive all the data sent, but not necessarily all at once. You may as well receive pieces of data.
Your approach of sending the length is hence valid: once you've received the length, you cann then load a buffer, if needed accross successive reads, until you got everything that you expected. So you have to loop on receives, and define a strategy on how to ahandle extra bytes received.
Datagramme (packet oriented) sockets:
If your application is really packet oriented, you may consider to create a datagramme socket, by requesting linux or windows socket(), the SOCK_DGRAM, or better SOCK_SEQPACKET socket type.
Risk with your binary size data:
Be aware that the way you send and receive your size data appers to be assymetric. You have hence a major risk if the sending and receiving between machine with CPU/architectures that do not use the same endian-ness. You can find here some hints on how to ame your code platform/endian-independent.
TCP socket is a stream based, not packet (I assume you use TCP, as to send length of packet in data does not make any sense in UDP). Amount of bytes you receive at once does not have to much amount was sent. For example you may send 10 bytes, but receiver may receive 1 + 2 + 1 + 7 or whatever combination. Your code has to handle that, be able to receive data partially and react when you get enough data (that's why you send data packet length for example).
I am implementing both server and client side of a simple file download program. Client side requests file names from the server with get command, and server responses quickly. While server writes to the socket, clients reads the socket and prints out the buffer. After that time, program starts not to interpret my commands unless I press 'Enter' twice. (You can see it from Shell output below)
After debugging, I found out that it is because of the buffer size. While server writing to the socket; everything works properly if I use a small buffer size. But if I use a larger buffer size such as 1024, that problem occurs. How can I get rid of this issue?
#define F_BUFF_SIZE 1024
On server side:
/* ... */
if(!strcmp(buffer, "list\n")) {
char buff[F_BUFF_SIZE];
bzero(buff, F_BUFF_SIZE);
pt_ret = pthread_create(&thread_id, NULL, getfiles, (void*) buff);
pthread_join(thread_id, pt_ret);
n = write(sock, buff, F_BUFF_SIZE);
/* ... */
On client side:
/* ... */
char buffer[F_BUFF_SIZE];
bzero(buffer, F_BUFF_SIZE);
n = read(b_sock, buffer, F_BUFF_SIZE - 1);
if (n < 0) {
#ifdef _DEBUG_
fprintf(stderr, "Error: Could not read from the socket.\n");
#endif
return 0;
}
fputs(buffer, stdout);
/* ... */
Shell:
Opening socket: OK!
Connecting: OK!
# list
client
project1.mk
cs342.workspace
client.c
project1.project
cs342.workspace.session
server
cs342_wsp.mk
server.c
cs342.tags
# get
# take
get
# take
Unknown command.
...
There is no magic to having a smaller buffer size, this is just exposing that you have an error elsewhere. Joerg's comment is an important point - you need to be reading the same amount of data that you are writing. I'm wondering if there's also an issue with how you populate buff. You need to make sure that you are not overrunning the end of the buffer, or forgetting to add a null terminator to the end of the string.
By the way, it is important to read the size - 1; you're correct to do this since read won't append a null terminator to a string. You just need to be sure that you're writing that amount because otherwise there can be problems.
You must have some issue along these lines - this would explain why changing the size avoids the problem because problems like this are only exposed when the numbers line up perfectly. Running the program in valgrind may expose the issue (look for invalid write errors).
#ahmet, your useful question lead me to research a little bit. If you read a little bit on this paper you will have a better idea of what you are dealing with and then you will be able to determine the best buffer size in your situation.
As you may know, values like that should always be part of the settings of the application, so don't scramble those values inside the code.
Also, here are good advices on how to figure buffer size, hope that helps,