I would like to programmatically find where the video was paused and resumed during the entire recording.
I have a screen recording by OBS studio which is over 7 hours and it has many pause and resume. So I want to programmatically find times / frames / location of this pauses and resumes.
The file type is MKV
As suggested in the comments I don't think this will be straightforward unless you have some clock points etc.
You could use a video analysis solution to go trough the frame by frame - it will be processing intensive but may meet your needs.
For example, using openCV you could read in the video frame by frame and compare each frame with it's predecessor, identifying groups of frames which are mostly the same (assuming your video frames are all more or less the same when you are paused).
If the video includes some sort of play or pause button on the screen when it paused you could also use OpenCv to detect this.
Related
I'm using libao (ao_play) to play some buffers. I listen the keyboard keys and for each key I have a wav sound to play. It's simple.
With ao_play I see that the application blocks while is playing the sound. Because I want to play multiple audios at same time, I needed to use threads (with pthread lib).
It works, but I fell like a workaround and if I play to much files (maybe 10 or something like this) so everything stuck for some seconds and so come back.
Well, my question is: how to play multiple sounds at same time non-blocking using libao (and not using threads)?
This not a real design, more like a guess.
First of all, you'll need threads because it's a good old tradition to separate computations from visualisations, or audializations in this case. You'll need an audio thread that renders the stream and sends it to the output.
So, each time your main thread discovers a keypress, it sends a note to the audio thread. That latter captures an event and adds a wave to the currently played stream. The stream is rendered in frames (64, or 1024, or 10240 samples, or whatever you fancy your latency, if the wave itself is a simple mix of few possible samples, it can be notably realtime.) You should keep track of notes currently played, position per each sample. If latency is low, thus granularity high, you can even align sample edges by buffer edges, which would notably simplify rendering.
And after current buffer is rendered you simply send it to DAC and proceed with the next frame.
A quick glance at libao's help page does not reveal any mixing capabilities, so you'll need to create a simple mixer on your own, or you may actually need an existing solution, some simple opensource audio rendering library.
I am making a game for the GameBoy in GBDK, and I'm trying to add sounds to the game. GBDK has a function that plays sounds from an array of values, the only problem is that while its playing the sound the rest of the script freezes. Is there a way I can get them to run at the same time?
There is no way to have code running while using sampled audio playback. This is due to the fact that it actually uses full CPU to preform this playback. If you want to use regular sound effects, you'll either need to pause the game while they play, or use a different method. I'll try to summarize using the other playback method below, but it is kind of complicated and I'm no expert.
Using "normal" sound effects
This is kind of WIP - I'm not too experienced with it but it should let you get started.
To use sound effects, you need to write to GameBoy audio registers. This is found in GBDK's hardware.h, which is automatically included with references to gb\gb.h. But (of course) the registers don't have any documentation. This information is found on the GB Cribsheet. There's also this sound documentation file (unfortunately it behaves weirdly on windows encodings - Open with something other than notepad), along with some other information found on the Devrs.com sound documentation.
Working off of GBSOUND.TXT:
The addresses through which the sound channels can be accessed are:
$Addresses: (Description), (Register shorthand)
$FF10 -- $FF14: Channel 1, Referred to as NR10-NR14
$FF15 is unused, was probably going to be a sweep reg for channel 2
originally
$FF16 -- $FF19: Channel 2, Referred to as NR21-NR24
$FF1A -- $FF1E: Channel 3, Referred to as NR30-NR34
$FF1F is unused, was probably going to be a sweep reg for channel 4
originally
$FF20 -- $FF23: Channel 4, Referred to as NR41-NR44
$FF24 controls the Vin status and volume, Referred to as NR50
$FF25 selects which output each channel goes to, Referred to as NR51
$FF26 is the status register, and also controls the sound circuit's power.
Referred to as NR52
$FF27 -- $FF2F are unused.
$FF30 -- $FF3F is the load register space for the 4-bit samples for channel
3
In GBDK, the registers are named NR10_REG, NR11_REG, NR12_REG, ect.
Also, try looking at the example program sound.c, which doesn't compile for me unfortuantely. I might edit this to include more info.
To answer #franklin's question:
Which begs the question, how does a gameboy play both the game and sound at the same time?
They usually don't do that with sample playback. For instance, if you look at Pokémon Yellow, Pikachu's cry is done with sample playback. But while that is playing, nothing else is done. On the other hand, things like normal background music are done using the other audio hardware (sorry, not very detailed wiki link). Similarly, while move sound effects are done with the noise channel (used for the sample playback as well), they aren't actually sampled audio. As such, the game can continue running.
I'm building an app in which I create a video.
Problem is, sometime (well... most of the time) the frame acquisition process isn't quick enough.
What I'm currently doing is to skip the current frame acquisition if I'm late, however FFMPEG/libavcodec considers every frame I pass to it as the next frame in line, so If I drop 1 out of 2 frames, a 20seconds video will only last 10. More problems come in as soon as I add sound, since sound processing is way faster...
What I'd like would be to tell FFMPEG : "last frame should last twice longer that originally intended", or anything that could allow me to process in real time.
I tried to stack the frames at a point, but this ends up killing all my memory (I also tried to 'stack' my frames in the hard drive, which was way to slow, as I expected)
I guess I'll have to work with the pts manually, but all my attempts have failed, and reading some other apps code which use ffmpeg, such as VLC, wasn't of a great help... so any advice would be much appreciated!
Thanks a lot in advance!
your output will probably be considered variable framerate (vfr), but you can simply generate a timestamp using wallclock time when a frame arrives and apply it to your AVFrame before encoding it. then the frame will be displayed at the correct time on playback.
for an example of how to do this (at least the specifying your own timestamp part), see doc/examples/muxing.c in the ffmpeg distribution (line 491 in my current git pull):
frame->pts += av_rescale_q(1, video_st->codec->time_base, video_st->time_base);
here the author is incrementing the frame timestamp by 1 in the video codec's timebase rescaled to the video stream's timebase, but in your case you can simply rescale the number of seconds since you started capturing frames from an arbitrary timebase to your output video stream's timebase (as in the above example). for example, if your arbitrary timebase is 1/1000, and you receive a frame 0.25 seconds since you started capturing, then do this:
AVRational my_timebase = {1, 1000};
frame->pts = av_rescale_q(250, my_timebase, avstream->time_base);
then encode the frame as usual.
Many (most?) video formats don't permit leaving out frames. Instead, try reusing old video frames when you can't get a fresh one in time.
Just an idea.. when it's lagging with the processing have you tried to pass to it the same frame again (and drop the current one)? Maybe it can process the duplicated frame quickly.
There's this ffmpeg command line switch -threads ... for multicore processing, so you should be able to do something similar with the API (though I have no idea how). This might solve your problem.
I'm going to write an application in Silverlight that consists of 2 threads, one that plays sound and another that records sound. And whatever is recorded will be what was played plus some ambient noise.
The problem is that Silverlight adds a delay to the sound to be played, and because I don't know how much is this delay, I would not know precisely what was played when something is recorded.
Do you know where I can find more information about this delay (how much is it, is it constant, will it change if I restart my application or computer, will it be the same in different computers, ...), or how could I measure it with an accuracy of 1 ms?
To measure the delay you can play some form of generated sound (like sin wave with increasing amplitudes), capture it and match input and output signals.
The delay itself is a complicated matter especially when dealing with low latencies. There are a lot of things involved in building delay including SL itself, audio stack, OS and audio hardware. Some additional info is here.
I'm trying to play only part of a sound using FMOD, say frames 50000-100000 of a 200000 frame file.
I have found a couple of ways to seek forward (i.e. to start playback at frame 50000) but I have not found a way to make sure the sound stops playing at 100000. Is there any way FMOD can natively do this without having to add lbsndfile or the like into the picture?
I should also mention that I am using the streaming option. I have to assume that these sounds are arbitrarily large and cannot be comfortably/quickly loaded into memory.
You can use Channel::setDelay for sample accurate starting and stopping of sounds. Use FMOD_DELAYTYPE_DSPCLOCK_START to set the start time of the sound and FMOD_DELAYTYPE_DSPCLOCK_END to set the end time.
Check out the docs for Channel::setDelay, FMOD_DELAYTYPE, System::getDSPClock.
You should be able to use the streaming callback to stop the stream when you get to the desired point.
Option 1: When you create the stream, set lenbytes to an even divisor of the number of frames you wish to play. In your example, set 'lenbytes' to 5000, then keep a counter in the callback. When you get to 10, stop the stream.
Option 2: use FSOUND_Stream_AddSyncPoint with pcmoffset set to your desired stopping point. Register a callback with FSOUND_Stream_SetSyncCallback. Stop the stream in the callback.
To start playback at sample 50,000 and end at 100,000 you could do the following assuming the sound file sample rate and the system sample rate are the same. As DSP clock works in system output samples you may need to do some maths to adjust your end sample in terms of output rate. See Sound::getDefaults for sound sample rate and System::getSoftwareFormat for system rate.
unsigned int sysHi, sysLo;
// ... create sound, play sound paused ...
// Seek the data to the desired start offset
channel->setPosition(50000, FMOD_TIMEUNIT_PCM);
// For accurate sample playback get the current system "tick"
system->getDSPClock(&sysHi, &sysLo);
// Set start offset to a couple of "mixes" in the future, 2048 samples is far enough in the future to avoid issues with mixer timings
FMOD_64BIT_ADD(sysHi, sysLo, 0, 2048);
channel->setDelay(FMOD_DELAYTYPE_DSPCLOCK_START, sysHi, sysLo);
// Set end offset for 50,000 samples from our start time, which means the end sample will be 100,000
FMOD_64BIT_ADD(sysHi, sysLo, 0, 50000);
channel->setDelay(FMOD_DELAYTYPE_DSPCLOCK_END, sysHi, sysLo);
// ... unpause sound ...