Der's blag

Der's blag

Open-source game streaming solution?

— Development

Update: ALVR exists, but unfortunately it’s only designed to stream games to VR glasses.

General problems:

  • There are no open-source game streaming solutions for Linux
  • There are multiple closed-source clients, but only one closed-source server (included in Steam)
  • Steam does not support VA-API for hardware acceleration, so streaming is CPU-intensive and not as fast as it could be.

Let’s go through all the problems:

Game streaming for Linux overview.

There is a great overview you can find on ArchWiki.

  • If you want to host games on Linux, your option is using Steam.
  • All the other options provide hosting only on Windows machines, or no hosting at all.

I have personally tried Playkey (cloud-hosting only).

Good things about it:

  • When it works, it works very smoothly only with minor stuttering.

Not very nice things about it:

  • The client is tricky to get working.
  • It runs under wine.
  • Sometimes I had to restart the streaming 10 times to get the game to start.
  • There is no way to change the language: I play from Germany, but I want Steam and games to be in English, so I had to change the language eveywhere in the beginning of each session.
  • There is no way to install mods.
  • You have to pay for the game, and then for the service.

Game streaming with Steam

I have used this client a lot, and I have a lot of good things to say about it:

  • It provides a really nice, smooth and stutter-free experience.
  • It is also very easy to set up: just start the client on two different computers, and instead of Play click Stream.

There are problems too:

  • It is closed-source, you cannot just tweak something if you really want to.
  • VA-API encoding acceleration is not supported.
  • Because of that, streaming might be very CPU-intensive.
  • This might be a problem for your system.
  • On my Ryzen 5 3600 system I could only get 48 fps on average (FullHD resolution).
  • That was a big suprise to me, because the experience was overall very nice.
  • Your home network should be pretty stable (wired, or 5GHz wireless).

Technical details

You can find your streaming log in ~/.steam/steam/logs/streaming_log.txt (on a server).

Looking at the log, I can instantly see:

  • All the video parameters: Desktop OpenGL NV12 + libx264 main (4 threads).
  • Average framerate of 48fps.
  • All the delays (around 18ms total).

Experiments

I’ve done some experiments, mostly based on this work.

Things I have discovered:

  • It is not easy to reach 1920x1080@60fps encoding in real time on my GPU (RX 580)
  • Using x11grab introduces a bottleneck. Using ksmgrab is much faster.
  • There are still unknown bottlenecks. I don’t get why I can only capture, for example, 40fps, but I can run 4 captures at the same time without any framerate penalty.
  • Is it limited by GPU clock speed?
  • Or maybe it is some kind of inefficiency in ffmpeg?
  • If I force high-performance mode on my GPU (echo high > /sys/class/drm/card0/device/power_dpm_force_performance_level), I can reach 60fps with h264_vaapi.

In the end, streaming command looks like this:

sudo ffmpeg -r 60 -device /dev/dri/card0 -f kmsgrab -i - -vaapi_device /dev/dri/renderD128 -vf 'hwmap=derive_device=vaapi,scale_vaapi=w=1920:h=1080:format=nv12' -rc_mode CBR -b:v 1M -c:v h264_vaapi -qp:v 24 -aspect 16:9 -f mpegts - | nc -v -u host post`
  • sudo is required for kmsgrab capture method.
  • -r 60 sets input framerate. For some reason, it falls back to 30fps otherwise.
  • -c:v h264_vaapi sets encoding codec. If you want to see more, check ffmpeg -h encoder=h264_vaapi.
  • -rc_mode CBR -b:v 1M sets constant-bitrate mode.
  • -qp:v 24 sets picture quality?
  • -f mpegts - | nc -v -u host port everything is then encoded as a stream and sent to host:port using UDP

Client command:

nc -u -vlp 9000 | mpv -quiet - --no-cache --untimed --no-demuxer-thread --vd-lavc-threads=1 --profile=low-latency --opengl-glfinish=yes --opengl-swapinterval=0 --framedrop=no --speed=1.01 -fps=60
  • nc -u -vlp 9000 | receives the network stream.
  • Bunch of other options to ensure close to zero latency. Some taken from there. I am not sure if that is the best combination.

Summary

In the same answer there has a following statement:

This achieves about 250ms latency for 1080p@60hz at around 3Mbps, which is ok for streaming shows over wifi. mpv can adjust for lip sync (CTRL +- during play). It’s tolerable for streaming desktop mouse/keyboard interactions for media control, but it’s unusable for real-time gaming (see NVidia Shield, Google Stadia for remote gaming)

I don’t think that is the case. From visual perspective, it looks even faster than steam remote play (so definetely suitable for gaming), but some crude tests show me that the delay is around 40-50ms (against ~20ms of steam).

It is at the same time very bad (should not be a very pleasant gaming experience) and very good (for a very crude solution).

If I to continue this project, I just have to hope that there are no unsolvable challenges here.

The next logical step: re-write capturing and encoding in Rust or C++. This would allow me to measure delays on capture and encoding steps.