# everything i learned about streaming video in linux
\-nemo

---

# demo

---

# let's talk about X11

```
+-------------------------+
|    input devices        |
|  keyboard | mouse       |
+-------^---+-^-----------+
        |     |
        |     |                 +------------------------+
      +-v-----v-------+-------->+                        |
      |               |         |                        |
      |   X Server    +-------->+                        |
      |               |         |                        |
      +--+---------+--+-------->+      Screen            |
         ^         ^            |                        |
         |         |            |                        |
         |         |            +------------------------+
         v         v
 +-------+---+  +--+-------+
 | browser   |  |          |
 | X client  |  | xterm    |
 |           |  |          |
 +-----------+  +----------+
```

---

# x11 - displays and screens

- You "select" a display with the `:$DISPLAY:SCREEN` syntax.

- So the first screen on your main display server would be :0.0

- You can set the display an application launches with DISPLAY environment variable

`export DISPLAY=:0.1 leafpad`

---


# Xvfb

>Xvfb or X virtual framebuffer is a display server implementing the X11 display server protocol. In contrast to other display servers, Xvfb performs all graphical operations in virtual memory without showing any screen output.

>However, no output is shown. This virtual server does not require the computer it is running on to have any kind of graphics adapter, a screen or any input device.

---

# `xwd`

> xwd - dump an image of an X window

```
xwd -root image.xwd
convert image.xwd image.png
```

---

# `ffmpeg`

can "grab" X11 windows

```
ffmpeg -f x11grab -framerate 10 -i :1.0 \
  -vaapi_device /dev/dri/renderD128 \
  -vf 'format=nv12,hwupload,scale_vaapi=w=854:h=480' \
  -c:v h264_vaapi -qp:v 26 -bf 0 -tune zerolatency -f mpegts \
  udp://localhost:12345
```

---

# video devices

- `/dev/video*`

```
v4l2-ctl  --device /dev/video2  --all
Driver Info:
  Driver name      : uvcvideo
  Card type        : HD Pro Webcam C920
  Bus info         : usb-0000:00:14.0-4
  Driver version   : 5.7.9
  Capabilities     : 0x84a00001
    Video Capture
    Metadata Capture
    Streaming
    Extended Pix Format
    Device Capabilities
  Device Caps      : 0x04a00000
    Metadata Capture
    Streaming
    Extended Pix Format
Priority: 2
Format Metadata Capture:
  Sample Format   : 'UVCH' (UVC Payload Header Metadata)
  Buffer Size     : 1024
```

---

# v4l2loopback

> v4l2loopback - a kernel module to create V4L2 loopback devices

>this module allows you to create "virtual video devices" normal (v4l2) applications will read these devices as if they were ordinary video devices, but the video will not be read from e.g. a capture card but instead it is generated by another application. this allows you for instance to apply some nifty video effects on your Skype video... it also allows some more serious things (e.g. I've been using it to add streaming capabilities to an application by the means of hooking GStreamer into the loopback devices).

---

# create new webcams!

```
modprobe v4l2loopback devices=4
```

`/dev/video3,4,5,6,7`

---

# ffmpeg can "write" to v4l

```
ffmpeg -f x11grab \
       -framerate 4 \
       -s 854x480 \
       -i :1.0 \
       -pix_fmt yuv420p \
       -f v4l2 /dev/video3
```

`ffmpeg -pix_fmts`

---

# aside: MJPEG over HTTP

`ffmpeg -i ... -f mjpg -o file.jpg`

```
Content-Type: multipart/x-mixed-replace; boundary="JPGBoundary"

--JPGBoundary
Content-Type: image/jpeg

<jpeg bytes>
--JPGBoundary
Content-Type: image/jpeg

<jpeg bytes>
--JPGBoundary
... and so on ...
--JPGBoundary--
```

---

# MJPEG streaming is simple

- `mjpeg-server` (uses `ffmpeg -f mjpg`)
- `ustreamer` (only supports `/dev/video*`)

---

```
+---------------------+        +---------------------------+
|                     |        |                           |
|     Xvfb Virtual    +------> |      ffmpeg -f x11grab    |
+---------+-----------+        |     -o /de^/video3        |
          ^                    +---------------------------+
   +------+----------+                     |
   |  X Server       +<---+      +---------+-----------+
   +---+-------------+    |      |  /dev/video3        |
       ^                  |      |                     |-???->USTREAMER->Viewers ???
   +---+---+              |      +-------+-------------+
   | baba  |              |              |
   | is    |        +-----+---------+    |   +-----------+
   | you   |        |               |    +-->+  Zoom     |
   |       |        |  xdotool      |        |           |
   +-------+        +---------------+        +-----------+
```

---

# security

YOLO!

Pitch: RFCs We Love!