Use case: I have two gaming PCs running a VR game in two separate rooms and a laptop in the living room connected to a projector. The VR headsets are wireless and will be in the same room as the projector with two people playing multiplayer together and the spectator screens of each gaming PC displayed side by side on the projector for others to watch the action. The game is a rhythm game (Beat Saber) so low latency is extremely important. On the other hand I can sacrifice on video quality because each desktop will only display on a 960x540 portion of the 1080p projector screen. Also audio is already taken care of and doesn't need to be transmitted to the laptop.
I have written a program with WPF and C# which displays two webpages side by side with a black bar at the top and bottom. The idea was to log into a low latency screen sharing webpage (for example parsec.app) and connect one of the PCs on each side. This actually works, but the problem is that after connecting the second computer, they both become very laggy and low fps when viewing content with a lot of movement. I really need both video streams to be smooth and with low latency. (<150ms) So using a third party service for the sharing/streaming of the screens seems to be out of the question. So I have to find a way to send the desktop streams directly over the LAN and then display them side by side with my program. I would like to have my own program display the streams so that I have authority over the layout and can add thematic pictures to the unused space on the screen. Maybe even make it customizable with up to four streams simultaneously in the future.
The laptop and one of the gaming PCs are only connected to the LAN via WiFi and the other gaming PC is connected via Ethernet. I have done some research and ffmpg or NDI seem to be the lowest latency ways to send video through a network, only I have no idea how to use them and don't have any experience programming network applications. I tried streaming my screen from one PC to the other with VLC using UDP but couldn't even get that working..
Here's a link to my visual studio project so far:
https://drive.google.com/file/d/1W7khWBvKZ1zMvreH9nyfAHPVQ6BDKN5Z/view?usp=share_link
Here's a video showing my program in action:
https://drive.google.com/file/d/1db3EHHV23mvdky36fcox9--crlrbZXK4/view?usp=share_link
Does anyone have any insights or ideas on how to solve my problem? Of course I'm willing to do some studying too if someone can point me to the resources where I can learn the required skills to make this work.
How to play sounds at different tones in C without using any external library? I know there are dozens of sound libraries in C that allows you to play sound but what I want to know is how does that work behind? How do you tell the computer to play a certain note at a certain tone/frequency?
I know it's possible on windows using the sound() function but I can't find any documentation talking about Linux, all that I found is the beep() function (or write(1, "\a", 1)) that outputs the default terminal beep but I can't figure out how to play different sounds.
The Linux kernel native audio API is ALSA (Advanced Linux Sound Architecture).
Example of raw audio playback with ALSA:
https://gist.github.com/ghedo/963382/815c98d1ba0eda1b486eb9d80d9a91a81d995283
However, ALSA is a low-level API that is not recommended to be used directly by higher-level applications.
A modern system audio API for GNU/Linux would be either PulseAudio (the current default on Ubuntu), or the newer and arguably better PipeWire (the default on Fedora).
Example of raw audio playback with PipeWire that generates audio "from scratch":
https://docs.pipewire.org/page_tutorial4.html
How do you tell the computer to play a certain note at a certain tone/frequency?
Sound is a mechanical vibration that propagates through the air (or another medium). It can be represented digitally as a sequence of numerical values representing air pressure at a given sampling rate. To play a given tone/frequency, generate a sine wave of that frequency (at the playback sampling rate) and use the sound API of your choice to play it.
See the PipeWire tutorial above for an example generating a 440Hz tone.
About PulseAudio/PipeWire:
These libraries are typically part of the OS and exposed as system APIs (so they are not "external libraries" if that means some library to ship with your program or to ask users to install), and they should be used by applications to play audio.
Behind the scene, these libraries handle audio routing, mixing, echo-canceling, recording, and playback to the kernel through ALSA (or using Bluetooth, etc). Everything that users and developers expect from the system audio layer.
Until recently, PulseAudio was the de-facto universal desktop system audio API, and many apps still use the PulseAudio API to play audio on GNU/Linux.
PipeWire includes compatibility with PulseAudio, so that apps using the PulseAudio API will keep working in the foreseeable future.
Example of raw audio playback with PulseAudio:
https://freedesktop.org/software/pulseaudio/doxygen/pacat-simple_8c-example.html
I hope to know how to capture audio by using dummy sound card driver.
I'm thinking how to implement the steps below.
we play audio in ubuntu, however the audio is just played through dummy sound card driver, to capture audio stream.
captured audio is sent to windows through network.
audio is actually played in windows.
What you need is to activate ALSA snd-aloop module, that provides a full-duplex virtual loopback soundcard. Please have a look to the following links for instructions about activation and example usage:
https://github.com/TheSalarKhan/Linux-Audio-Loopback-Device
https://sysplay.in/blog/linux/2019/06/playing-with-alsa-loopback-devices/
A couple of important points to consider:
The subdevices are linked in pairs; whatever you play on hw:n,0,m goes out on hw:n,1,m (see the example in the 1st link)
The first application opening one of the subdevices will force the second application to use the same set of parameters: sample rate, format, number of channels. For example, suppose the recording application opens a capture stream on hw:2,1,0 with stereo/44100/S16_LE format; the playback application on hw:2,1,0 will be forced to use a the same stereo/44100/S16_LE format
Hope this helps
I have a bare-metal application running on a tiny 16 bit microcontroller (ST10) with 10BASE-T Ethernet (CS8900) and a Tcp/IP implementation based upon the EasyWeb project.
The application's main job is to control a led matrix display for public traffic passenger information. It generates display information with about about 41 fps and configurable display size of e.g. 160 × 32 pixel, 1 bit color depth (each led can be just either on or off).
Example:
There is a tiny webserver implemented, which provides the respective frame buffer content (equals to led matrix display content) as PNG or BMP for download (both uncompressed because of CPU load and 1 Bit color depth). So I can receive snapshots by e.g.:
wget http://$IP/content.png
or
wget http://$IP/content.bmp
or put appropriate html code into the controller's index.html to view that in a web browser.
I also could write html / javascript code to update that picture periodically, e.g. each second so that the user can see changes of the display content.
Now for the next step, I want to provide the display content as some kind of video stream and then put appropriate html code to my index.html or just open that "streaming URI" with e.g. vlc.
As my framebuffer bitmaps are built uncompressed, I expect a constant bitrate.
I'm not sure what's the best way to start with this.
(1) Which video format is the most easy to generate if I already have a PNG for each frame (but I have that PNG only for a couple of milliseconds and cannot buffer it for a longer time)?
Note that my target system is very resource restricted in both memory and computing power.
(2) Which way for distribution over IP?
I already have some tcp sockets open for listening on port 80. I could stream the video over HTTP (after received) by using chunked transfer encoding (each frame as an own chunk).
(Maybe HTTP Live Streaming doing like this?)
I'd also read about thinks like SCTP, RTP and RTSP but it looks like more work to implement this on my target. And as there is also the potential firewall drawback, I think I prefer HTTP for transport.
Please note, that the application is coded in plain C, without operating system or powerful libraries. All stuff is coded from the scratch, even the web server and PNG generation.
Edit 2017-09-14, tryout with APNG
As suggested by Nominal Animal, I gave a try with using APNG.
I'd extend my code to produce appropriate fcTL and fdAT chunks for each frame and provide that bla.apng with HTTP Content-Type image/apng.
After downloading those bla.apng it looks useful when e.g. opening in firefox or chrome (but not in
konqueror,
vlc,
dragon player,
gwenview).
Trying to stream that apng works nicely but only with firefox.
Chrome wants first to download the file completely.
So APNG might be a solution, but with the disadvantage that it currently only works with firefox. After further testing I found out, that 32 Bit versions of Firefox (55.0.2) crashing after about 1h of APNG playback were about 100 MiB of data has been transfered in this time. Looks that they don't discard old / obsolete frames.
Further restrictions: As APNG needs to have a 32 bit "sequence number" at each animation chunk (need 2 for each frame), there might to be a limit for the maximum playback duration. But for my frame rate of 24 ms this duration limit is at about 600 days and so I could live with.
Note that APNG mime type was specified by mozilla.org to be image/apng. But in my tests I found out that it's a bit better supported when my HTTP server delivers APNG with Content-Type image/png instead. E.g. Chromium and Safari on iOS will play my APNG files after download (but still not streaming). Even the wikipedia server delivers e.g. this beach ball APNG with Content-Type image/png.
Edit 2017-09-17, tryout with animated GIF
As also suggested by Nominal Animal, I now tried animated GIF.
Looks ok in some browsers and viewers after complete download (of e.g. 100 or 1000 frames).
Trying live streaming it looks ok in Firefox, Chrome, Opera, Rekonq and Safari (on macOS Sierra).
Not working Safari (on OSX El Capitan and iOS 10.3.1), Konqueror, vlc, dragon player, gwenview.
E.g. Safari (tested on iOS 10.3.3 and OSX El Capitan) first want to download the gif completely before display / playback.
Drawback of using GIF: For some reason (e.g. cpu usage) I don't want to implement data compression for the generated frame pictures. For e.g. PNG, I use uncompressed data in IDAT chunk and for a 160x32 PNG with 1 Bit color depth a got about 740 Byte for each frame. But when using GIF without compression, especially for 1 Bit black/white bitmaps, it blows up the pixel data by factor 3-4.
At first, embedded low-level devices not very friendly with very complex modern web browsers. It very bad idea to "connect" such sides. But if you have tech spec with this strong requirements...
MJPEG is well known for streaming video, but in your case it is very bad, as requires much CPU resources and produces bad compression ratio and high graphics quality impact. This is nature of jpeg compression - it's best with photographs (images with many gradients), but bad with pixel art (images with sharp lines).
Looks that they don't discard old / obsolete frames.
And this is correct behavior, since this is not video, but animation format and can be repeated! Exactly same will be with GIF format. Case with MJPEG may be better, as this is established as video stream.
If I were doing this project, I would do something like this:
No browser AT ALL. Write very simple native player with winapi or some low-level library to just create window, receive UDP packet and display binary data. In controller part, you must just fill udp packets and send it to client. UDP protocol is better for realtime streaming, it's drop packets (frames) in case of latency, very simple to maintain.
Stream with TCP, but raw data (1 bit per pixel). TCP will always produce some latency and caching, you can't avoid it. Same as before, but you don't need handshaking mechanism for starting video stream. Also, you can write your application in old good technologies like Flash and Applets, read raw socket and place your app in webpage.
You can try to stream AVI files with raw data over TCP (HTTP). Without indexes, it will unplayable almost everywhere, except VLC. Strange solution, but if you can't write client code and wand VLC - it will work.
You can write transcoder on intermediate server. For example, your controller sent UDP packets to this server, server transcode it in h264 and streams via RTMP to youtube... Your clients can play it with browsers, VLC, stream will in good quality upto few mbits/sec. But you need some server.
And finally, I think this is best solution: send to client only text, coordinates, animations and so on, everything what renders your controller. With Emscripten, you can convert your sources to JS and write exact same renderer in browser. As transport, you can use websockets or some tricks with long-long HTML page with multiple <script> elements, like we do in older days.
Please, tell me, which country/city have this public traffic passenger information display? It looks very cool. In my city every bus already have LED panel, but it just shows static text, it's just awful that the huge potential of the devices is not used.
Have you tried just piping this through a websocket and handling the binary data in javascript?
Every websocket frame sent would match a frame of your animation.
you would then take this data and draw it into an html canvas. This would work on every browser with websocket support - which would be quite a lot - and would give you all the flexibility you need. (and the player could be more high end than the "encoder" in the embedded device)
I am working on an indoor vision based navigation in a GPS denied environment, and I'm using a 3DR Solo. I want to be able to send movement commands to the Solo through the DroneKit API. Are these mavlink commands supported for modes other than GUIDED and AUTO, or is channel overriding my only option?
It depends on how you want to control it. If you want to control the position of the copter, you will need to make hardware modifications to the solo, because it does not have the right sensors to be able to hold position indoors. If you want to control the raw angle of the copter, the cleanest way to do this is to send a SET_ATTITUDE_TARGET message. The problem is that this is only supported in guided mode, and guided mode won't arm without gps (even though low-level control like SET_ATTITUDE_TARGET doesn't rely on GPS). Guided-NoGPS mode is currently on the todo list for ArduCopter, so comment on the issue here to let the devs know that you are interested in this feature.