I need to display stream video using MediaElement in Windwso Phone application.
I'm getting from web-service a stream that contains frames in H264 format AND raw-AAC bytes (strange, but ffmpeg can parse with -f ac3 parameter only).
So, if try to play only one of stream (audio OR video) it plays nice. But I have issues when try it both.
For example, if I report video sample without timestamp and report audio with timestamp, my video plays 3x-5x faster then I need.
MediaStreamSample msSamp = new MediaStreamSample(
_videoDesc,
vStream,
0,
vStream.Length,
0,
_emptySampleDict);
ReportGetSampleCompleted(msSamp);
From my web-service I getting a DTS and PTS for video and audio frames in following format:
120665029179960
but when I set it for sample, my audio stream plays too slow and with delays.
Timebase is 90khz.
So, could someone tell me how I can resolve it? Maybe I should calculate others timestamps for samples? If so, show me the way, please.
Thanks.
Okay, I solved it.
So, what I need to do for sync A/V:
Calculate right timestamps for each video and audio frames using framerate.
For example, for video I have 90 kHz and for audio 48 kHz and 25 frames per second - my frame increments will be:
_videoFrameTime = (int)TimeSpan.FromSeconds((double)0.9 / 25).Ticks;
_audioFrameTime = (int)TimeSpan.FromSeconds((double)0.48 / 25).Ticks;
And now we should add these values for each sample:
private void GetAudioSample()
{
...
/* Getting sample from buffer */
MediaStreamSample msSamp = new MediaStreamSample(
_audioDesc,
audioStream,
0,
audioStream.Length,
_currentAudioTimeStamp,
_emptySampleDict);
_currentAudioTimeStamp += _audioFrameTime;
ReportGetSampleCompleted(msSamp);
}
For gettign video frame method will be the same with a _videoFrameTime incrementation instead.
Hope this will be helpfull for someone.
Roman.
Related
I'm using the Ant Media SDK to develop a webRTC capable app. The thing is that I need to send some frames every 300 ms to a recognition service. I have found that there's a listener that pass the frames:
surfaceTextureHelper.startListening((VideoFrame frame) -> {
//some code
}
The thing is, how can I get a byte[] from that frame, in Camera1API you had setPreviewCallback and received directly a byte[] with the frame. How can I make the same conversion??
Thanks in advance
I'm using FFmpeg library to receive and decode H.264/MPEG-TS over UDP with minimal latency (something MediaElement can't handle).
On a dedicated FFmpeg thread, I'm pulling PixelFormats.Bgr32 video frames for display. I've already tried InteropBitmap:
_section = CreateFileMapping(INVALID_HANDLE_VALUE, IntPtr.Zero, PAGE_READWRITE, 0, size, null);
_buffer = MapViewOfFile(_section, FILE_MAP_ALL_ACCESS, 0, 0, size);
Dispatcher.Invoke((Action)delegate()
{
_interopBitmap = (InteropBitmap)Imaging.CreateBitmapSourceFromMemorySection(_section, width, height, PixelFormats.Bgr32, (int)size / height, 0);
this.Source = _interopBitmap;
});
And then per frame update:
Dispatcher.Invoke((Action)delegate()
{
_interopBitmap.Invalidate();
});
But performance is quite bad (skipping frames, high CPU usage etc).
I've also tried WriteableBitmap: FFmpeg is placing frames in _writeableBitmap.BackBuffer and per frame update:
Dispatcher.Invoke((Action)delegate()
{
_writeableBitmap.Lock();
});
try
{
ret = FFmpegInvoke.sws_scale(...);
}
finally
{
Dispatcher.Invoke((Action)delegate()
{
_writeableBitmap.AddDirtyRect(_rect);
_writeableBitmap.Unlock();
});
}
Experiencing almost the same performance issues (tested with various DispatcherPriority).
Any help will be greatly appreciated.
I know it is too late, but I write this answer for those folks who are struggling to solve this problem.
Recently, I have done a rendering project using InteropBitmap in which I was able to run about 16 media player components in a WPF window at the same time on a core i7 1.6 Ghz CPU +8Gb RAM Laptop, with 25fps.
Here are the tips I took for performance tweaking:
First of all, I did not let GC handle my video packets. I Allocated memory using Marshal.AllocateHGlobal wherever I needed to instantiate a video frame and Disposed using Marshal.FreeHGlobal as soon as I did the rendering.
Secondly, I created a dispatcher thread for each individual media player. For more information, read "https://blogs.msdn.microsoft.com/dwayneneed/2007/04/26/multithreaded-ui-hostvisual/".
Thirdly, for aspect ratio, and generally the resizing purposes, I used native EmguCV library. This library helped me a lot on performance rather than using bitmaps and overlays and etc.
I think these steps help everyone that needs to render manually using InteropBitmap or etc.
We hav developed an application for video conference using silverlight.
It works properly for 15 to 19 min then video get stopped and silverlight plugin has crashed.
for video encoding we r using the JPEG encoder and single image from capturesource get encoded and send on each tick of timer..
I also tried to use Silversuite but message popup arrives i.e. Silversuite expire
Is der proper solution for encoding or timer or plug in...
Thanx...
we extend the crashing period from 15 min to 1 to 1 n 1/2 hr ......by flushing the mem stream and decreasing the receiving buffer size .....
I'm working on a WP7.1 app that records audio and plays it back. I'm using a MedialElement to playback audio. The MediaElement works fine for playing MP4 (actually M4A files renamed) downloaded from the server. However, when I try to play a recorded file with or without the WAV RIFF header (PCM in both cases) it does not work. It gives me an error code 3001, which I cannot find the definition for anywhere.
Can anyone point me to some sample code on playing recorded audio in WP7.1 that does not use the SoundEffect class. Don't want to use the SoundEffect class because it's meant for short audio clips.
This is how I load the audio file:
using (IsolatedStorageFile storage = IsolatedStorageFile.GetUserStoreForApplication())
{
using (Stream stream = storage.OpenFile(audioSourceUri.ToString(), FileMode.Open))
{
m_mediaElement.SetSource(stream);
}
}
This playing code looks good. Issue have to be in storing code. BTW 3001 means AG_E_INVALID_FILE_FORMAT.
I just realized that the "Average bytes per second" RIFF header value was wrong. I was using the wrong value for the Bits per Sample value, which should've been 16 bit since the microphone records in 16-bit PCM.
I want to write MJPEG picture internet stream viewer. I think getting jpeg images using sockets it's not very hard problem. But i want to know how to make accurate streaming.
while (1)
{
get_image()
show_image()
sleep (SOME_TIME) // how to make it accurate?
}
Any suggestions would be great.
In order to make it accurate, there are two possibilities:
Using framerate from the streaming server. In this case, the client needs to keep the same framerate (calculate each time when you get frame, then show and sleep for a variable amount of time using feedback: if the calculated framerate is higher than on server -> sleep more; if lower -> sleep less; then, the framerate on the client side will drift around the original value from server). It can be received from server during the initialization of streaming connection (when you get picture size and other parameters) or it can be configured.
The most accurate approach, actually, is using of timestamps from server per each frame (which is either taken from file by demuxer or generated in image sensor driver in case of camera device). If MJPEG is packeted into RTP stream, these timestamps are already in RTP header. So, client's task is trivial: show picture using time calculating from time offset, current timestamp and time base.
Update
For the first solution:
time_to_sleep = time_to_sleep_base = 1/framerate;
number_of_frames = 0;
time = current_time();
while (1)
{
get_image();
show_image();
sleep (time_to_sleep);
/* update time to sleep */
number_of_frames++;
cur_time = current_time();
cur_framerate = number_of_frames/(cur_time - time);
if (cur_framerate > framerate)
time_to_sleep += alpha*time_to_sleep;
else
time_to_sleep -= alpha*time_to_sleep;
time = cur_time;
}
, where alpha is a constant parameter of reactivity of the feedback (0.1..0.5) to play with.
However, it's better to organize queue for input images to make the process of showing smoother. The size of queue can be parametrized and could be somewhere around 1 sec time of showing, i.e. numerically equal to framerate.