My Android app uses MediaProjection to take screenshots and record a video at the same time.
The ImageReader is used for taking screenshots when necessary, the MediaRecorder records the screen into a mp4 file.
The screenshot part is done by using
private val imageReader = ImageReader.newInstance(imageSize.width, imageSize.height, PixelFormat.RGBA_8888, 2)
private val virtualDisplay = mediaProjection.createVirtualDisplay(
"ScreenCapture",
imageSize.width, imageSize.height, screenDensity,
0, imageReader.surface, null, null
)
The video recording part is using
private val mediaRecorder = MediaRecorder().apply {
setVideoSource(MediaRecorder.VideoSource.SURFACE)
val profile = CamcorderProfile.get(CamcorderProfile.QUALITY_HIGH)
setOutputFormat(profile.fileFormat)
setVideoEncoder(profile.videoCodec)
setVideoEncodingBitRate(profile.videoBitRate)
setVideoFrameRate(profile.videoFrameRate)
setVideoSize(imageSize.width, imageSize.height)
setOutputFile(storageProvider.recordingFileDescriptor.fileDescriptor)
prepare()
}
private val virtualDisplay: VirtualDisplay = mediaProjection.createVirtualDisplay(
"ScreenRecord",
imageSize.width, imageSize.height, screenDensity,
0, mediaRecorder.surface, null, null
)
So the same MediaProjection is used to fill an ImageReader surface and a MediaRecorder surface at the same time.
Users with Android 13 now report that only one of the surfaces will be filled. If the video recording is started after the screenshot service, the screenshot service will not receive any new images through imageReader.acquireLatestImage().
For Android 7 to 12 users, this still worked fine. They only saw this behavior when another app started MediaProjection.
Is there any way to use the same VirtualDisplay for both the ImageReader and the MediaRecorder? I can't think of any other solution.
The only existing solutions I can find are for the Camera2 API.
Related
At the page https://www.wowza.com/docs/how-to-build-a-basic-app-with-gocoder-sdk-for-ios there are the following examples:
if (self.goCoder != nil) {
// Associate the U/I view with the SDK camera preview
self.goCoder.cameraView = self.view;
// Start the camera preview
[self.goCoder.cameraPreview startPreview];
}
// Start streaming
[self.goCoder startStreaming:self];
// Stop the broadcast that is currently running
[self.goCoder endStreaming:self];
The equivalent Java code for Android is reported at the page https://www.wowza.com/docs/how-to-build-a-basic-app-with-gocoder-sdk-for-android#start-the-camera-preview, it is:
// Associate the WOWZCameraView defined in the U/I layout with the corresponding class member
goCoderCameraView = (WOWZCameraView) findViewById(R.id.camera_preview);
// Start the camera preview display
if (mPermissionsGranted && goCoderCameraView != null) {
if (goCoderCameraView.isPreviewPaused())
goCoderCameraView.onResume();
else
goCoderCameraView.startPreview();
}
// Start streaming
goCoderBroadcaster.startBroadcast(goCoderBroadcastConfig, this);
// Stop the broadcast that is currently running
goCoderBroadcaster.endBroadcast(this);
The code is self-explaining: the first blocks start a camera preview, the second blocks start a streaming and the third blocks stop it. I want the preview and the streaming inside a Codename One PeerComponent, but I didn't remember / understand how I have to modify both these native code examples to return a PeerComponent to the native interface.
(I tried to read again the developer guide but I'm a bit confused on this point).
Thank you
This is the key line in the iOS instructions:
self.goCoder.cameraView = self.view;
Here you define the view that you need to return to the peer and that we can place. You need to change it from self.view to a view object you create. I think you can just allocate a UIView and assign/return that.
For the Android code instead of using the XML code they use there you can use the WOWZCameraView directly and return that as far as I can tell.
I'm new to using codename one and I can not understand how we can take a picture from the camera using captureImage (); from the camerakit library.
I know it's possible with the Capture API (Capture.capturePhoto ();) but this library uses an application to take the photo and I want to do this directly
I created a button :
FloatingActionButton capture_button =
FloatingActionButton.createFAB(FontImage.MATERIAL_CAMERA);
capture_button.bindFabToContainer(hi, CENTER, BOTTOM);
capture_button.addActionListener(e -> {
ck.captureImage();
.............
and after that I tried to get my picture from the onImage function but it does not work.
#Override
public void onImage(CameraEvent ev) {
try {
byte[] jpegData = ev.getJpeg();
String str = new String(jpegData);
InputStream stream = FileSystemStorage.getInstance().openInputStream(jpegData);
OutputStream out = Storage.getInstance().createOutputStream("MyImage.jpg");
Util.copy(stream, out);
Util.cleanup(stream);
Util.cleanup(out);
StorageImage out = StorageImage.create("MyImage.jpg", jpegData, -1, -1);
............................
}
the byte array is empty. Help please.
Camera Kit broke a bit after its release due to changes in Camera Kit which is still not 1.0 level. This is tracked in this issue. Camera kit was supposed to reach 1.0 status months ago but still hasn't reached that point. We
are waiting for it to be at 1.0 level so we can make fixes against a stable version.
We also need a bit of time/resources to do that work which is something we are sorely lacking.
My app features a button to record audio (and another to play it back when the recording is over). I send the recorded audio files on a server. On Android the files is recorded as .amr (mime type audio/amr) and can be played back.
On iOS however the file can neither be played back on the device (iPhone 4 or 4S) nor on a computer. ffmpeg -i reports
[mov,mp4,m4a,3gp,3g2,mj2 # 0x2fac120] moov atom not found
9gMjOnnmsj9JJZR3.m4a: Invalid data found when processing input
Please note that VLC cannot play it either.
I give the m4a extension because Voice recorder uses it (along with aac codec).
Here is the code I use (mostly based on https://github.com/codenameone/CodenameOne/blob/master/Ports/iOSPort/src/com/codename1/impl/ios/IOSImplementation.java#L2768-L2794 ) :
audioMemoPath = ParametresGeneraux.getWRITABLE_DIR() + "AudioMemo-"
+ System.currentTimeMillis() +
(Display.getInstance().getPlatformName().equals("and")
? ".amr"
: ".m4a");
audioMemoMimeType = MediaManager.getAvailableRecordingMimeTypes()[0];
audioMemoRecorder = MediaManager.createMediaRecorder(audioMemoPath, audioMemoMimeType);
// If the permission audio has not been granted
// the audiomemorecoreder will be null
if (audioMemoRecorder != null) {
audioMemoRecorder.play();
boolean b = Dialog.show("Recording", "", "Save", "Cancel");
audioMemoRecorder.pause();
audioMemoRecorder.cleanup();
...
}
Moreover if I display the available mime types on iOS, it yields "audio/amr" which I doubt according to all the posts I could read that tell you amr is not supported on iOS. Looking at the source it appears amr is the by default mime type because it is always returned :
/**
* Gets the available recording MimeTypes
*/
public String [] getAvailableRecordingMimeTypes(){
return new String[]{"audio/amr"};
}
So my question : is it possible to record audio on iOS, and if it is, how can it be done ?
Any help appreciated,
Have you looked at the Capture class? That seems to be more straightforward.
https://www.codenameone.com/javadoc/index.html
Ok I got it working by overloading some methods of the MediaManager, namely getAvailableRecordingMimeTypes() and also createMediaRecorder() to prevent it from using its getAvailableRecordingMimeTypes method.
Here is the code for getAvailableRecordingMimeTypes():
/**
* Normally the method returns amr even for ios. So we overload the original
* method to return aac on ios.
* #return
*/
public static String[] getAvailableRecordingMimeTypes() {
if (Display.getInstance().getPlatformName().equals("ios")) {
return new String[]{"audio/aac"};
} else {
return new String[]{"audio/amr"};
}
}
createMediaRecorder() is left as is (copied without changes).
Now it is possible to record audio in iOS and play it back in both iOS and Android!
I have created an app for video demonstration using Codename one. I'm Facing some challenges when I'm running the app on my Google Android Phone as it does not allow a full screen view and also after the video is done playing, it does not go back or restart the video again. Another problem was that I had a button at the bottom at the borderlayout and each time I click the button, it corrupts the video and the video won't play anymore. These are codes used for my demonstration app Demonstration App 1, Demonstration App2 .
#Override
protected void postMain1(Form f) {
final MediaPlayer mp = findMpPresent();
try {
InputStream is = Display.getInstance().getResourceAsStream(getClass(), "/sbuda.mp4");
if (is != null) {
mp.setDataSource(is, "video/mp4", null);
} else {
}
} catch (IOException ex) {
ex.getMessage();
}
}
This is a bit unclear since I can't see the stop/start etc with a GUI builder application.
You can use native on-device controls for playback using setFullScreen. Notice that this works nicely on the device but has no equivalent on the simulator.
Once playback is finished the media no longer exists as your input stream has been depleted. You will need to create a new Media object. You can use the completion callback (the Runnable argument) to detect the end of the media.
I am trying to find solution to play YouTube video from http stream. Not as embedded player or via website. The reason of that as I need to play some clips locally in app that prevented to be embedded.
So the first task to obtain http stream is easy to solve, e.g. using YouTubeExtractor. For example from this youtube url
https://www.youtube.com/watch?v=31crA53Dgu0
YouTubeExtractor extracts such url for downloading video. As you can see there is no piece of path to concrete .mp4 or .mov file. There is also binding to IP, so the url won't work on your side.
https://r3---sn-puu8-3c2e.googlevideo.com:443/videoplayback?sparams=dur,gcr,id,initcwndbps,ip,ipbits,itag,lmt,mime,mm,mn,ms,mv,pl,ratebypass,requiressl,source,upn,expire&source=youtube&initcwndbps=3147500&pl=19&upn=oYYkSNBwoc8&fexp=9412913,9416126,9416891,9422596,9428398,9429016,9431012,9431364,9431901,9432206,9432825,9433096,9433424,9433623,9433946,9434085,9435504,9435703,9435937&id=o-AGanAaajMgOmp4VrXIwo9jewJnqlsvZecDxCcRpSN3lS&expire=1462821352&gcr=ua&ip=12.34.56.149&lmt=1458705532562546&ms=au&mt=1462799507&mv=m&dur=217.849&sver=3&itag=22&key=yt6&mn=sn-puu8-3c2e&mm=31&ipbits=0&mime=video/mp4&ratebypass=yes&requiressl=yes&signature=A142619EBA90D00CC46FF05B9CF1E3469B0EF196.072B532670A4D833582F94CF9C46F1F7D298F230&fallback_host=tc.v1.cache5.googlevideo.com
private string ExtractVideoUrl(string youtubeUrl)
{
IEnumerable<VideoInfo> videoInfos = DownloadUrlResolver.GetDownloadUrls(youtubeUrl, false);
var video = videoInfos.First();
if (video.RequiresDecryption)
{
DownloadUrlResolver.DecryptDownloadUrl(video);
}
return video.DownloadUrl; // videoInfos.First().DownloadUrl;
}
Then to download video this lib uses the code below. The interesting moment is in reading stream and writing to file the contents of youtube video.
var request = (HttpWebRequest)WebRequest.Create(this.Video.DownloadUrl);
if (this.BytesToDownload.HasValue)
{
request.AddRange(0, this.BytesToDownload.Value - 1);
}
// the following code is alternative, you may implement the function after your needs
using (WebResponse response = request.GetResponse())
{
using (Stream source = response.GetResponseStream())
{
**// HOW to Play source stream???**
using (FileStream target = File.Open(this.SavePath, FileMode.Create, FileAccess.Write))
{
var buffer = new byte[1024];
bool cancel = false;
int bytes;
int copiedBytes = 0;
while (!cancel && (bytes = source.Read(buffer, 0, buffer.Length)) > 0)
{
target.Write(buffer, 0, bytes);
....
So the general question is how to play video from that open stream? For example Chrome, Firefox and KMPlayer are doing this easily. The browsers generate a simple page with video tag, so it will be trivial to manage the player through JS. But...internal WebBrowser control can't, it suggests to download file. I tried CEFSharp (Chrome Embeded Framework) and no luck there. Maybe anybody know good video player WPF library which can streaming video? I also tried VLC.wpf and Shockwave Flash, but still unhappy with this extracted url. A lot of searching but results insufficient for now.
After 2 days researching I found the solution of how to play youtube video url directly from WPF application.
Some notes before:
WebBrowser component for WPF and the same for WinForms does not support HTML5. So no way to play video using <video> tag there
Chrome Embeded Framework (CEF)and its CEFSharp wrapper does not support <audio> and <video> tags by default. As I found on SO it is needed to recompile CEF to support audio and it seems video too. --enable-multimedia-streams option didn't work on my side at all to play video
WPF MediaElement component does not support playing video from streams. However there are workaroung exists, they are unsuitable to play youtube url's.
No .NET media player found which could play video from Stream object returned by HttpWebRequest
So the solution is in using GeckoFx webbrowser which is also available via nuget. As it is WinForms based component it is needed to use WinFormsHosting to use it in WPF. Final solution is:
<xmlns:gecko="clr-namespace:Gecko;assembly=Geckofx-Winforms">
....
<WindowsFormsHost Width="400" Height="300" HorizontalAlignment="Stretch" VerticalAlignment="Stretch" >
<gecko:GeckoWebBrowser x:Name="_geckoBrowser"/>
</WindowsFormsHost>
And simple code to use:
public void PlayVideo(string videoId)
{
var url = string.Format("https://www.youtube.com/watch?v={0}", videoId);
var videoUrl = ExtractVideoUrl(url);
_geckoBrowser.Navigate(videoUrl);
}
GeckoFx is depeneded on XulRunner runtime sdk. It is installed with nuget package though. One cons in this solution is the size of all app dependencies increased for over +70mb.