How crazy is my idea about WebRTC collabing tool?

[FarleyCZ]

Alright folks, I know that this kind of post is almost banned here, but hear me out. Especially in this day and age, when everyone is at home, there is a big demand for ways how to collaborate online. I have figured out some ways with remote desktop softwares that work fairly well, but they need kinda complex setup and some kind of audio loopback solution. (Either fancy RME/Motu card or some in-between driver software.) So I've been doing some research on how to streamline this into a simple product, that could be used more easily.

I've stumbled upon WebRTC that alows a fairly simple "semi-p2p" media connection between browsers. That's awesome for streaming or video-conferencing, but it can't do remote desktop due to browser's lack of ability to control mouse and keyboard. You need native code for that. Also getting audio from ASIO drivers into a browser is impossible on it's own.

But then I realized, wait a minute. A plugin is a bit of native code. It might be able to communicate with javascript running inside of a browser through localhost loop. The plugin might pass audio to the JS app through localhost and recieve commands from JS app through it too. And as it's a native bit of code, it might communicate with OS to execute those mouse movements and key strokes. Heck, it can even pass midi events.

That javascript app doesn't have to be in the browser at all. If we embedd a chromium instance into the plugin, it can actually call and run the JS app from the plugin itself. (But leaving it externally in Chrome would have advantages too. Smaller load in the DAW.)

On the side of your collaborator, he would run a JS client in his browser with a fairly straight forward video decoding WebRTC applet and a subrutine that would register his mouse clicks and keyboard strokes, it would feed those commands into the data channel of WebRTC and send it to the hosting site for aforementioned execution.

Here's a VERY LAME image I've just sketched up to help visualize this crazy idea:



So? How bonkers am I?

[BertKoor]

Link to image is broken. That I can prove.
(Edit: you fixed it)

Dunno how feasible your idea is, but me thinks sending mouse clicks is not the core problem. Network latency is...

[FarleyCZ]

I was able to run some remote-colalb setups with TeamViewer-ish software solutions and the latency wasn't that bad. Of course you won't get realtime keyboard jamming out of this concept. The commands will arive quickly, but the compression latencies will delay the audible feedback on the way back. (About 0.5s using available software now.) Still though. For sounddesign, arranging, mixing and overall nerding-out over one DAW it might be still very valuable.

Also WebRTC praise itself by using servers just for handshake and synchronisation. The actual data transfer should be peer2peer, so the latencies might be better than runing the whole thing through a server which all of the remote desktop solutions do now.