GUI Desktop

With docker-nvidia-glx-desktop or docker-nvidia-egl-desktop, users may start a GUI Desktop interface accelerated with NVIDIA GPUs. Both containers support OpenGL and Vulkan. Out of the two containers, docker-nvidia-glx-desktop is generally recommended to be used because it provides the best performance. However, docker-nvidia-egl-desktop is versatile in various environments and has less processes running, meaning less possible errors. It is also possible to be used in HPC clusters with Apptainer/Singularity available, and sharing a GPU with multiple containers is also possible.

Support is available in #ue4research:matrix.nrp-nautilus.io (use this link if you use a matrix.org account), just ask your questions there.

Please give the repositories a star!

DNS Setup

Use the command ping turn.nrp-nautilus.io on your client (install iputils-ping when using Linux if the ping command does not work) to check that your DNS is correctly configured. If the latency is considerably high (over 60-70 miliseconds), consider changing your client DNS server to 8.8.8.8 and 8.8.4.4 or the DNS over HTTPS / DNS over TLS options provided by Google. CloudFlare's DNS server (addresses 1.1.1.1 and 1.0.0.1) may show issues locating the nearest relay server, which is crucial for performance.

Usage

The below is a reference configuration xgl.yml for docker-nvidia-glx-desktop.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: xgl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: xgl
  template:
    metadata:
      labels:
        app: xgl
    spec:
      hostname: xgl
      containers:
      - name: xgl
        image: ghcr.io/selkies-project/nvidia-glx-desktop:latest
        env:
        - name: TZ
          value: "UTC"
        - name: SIZEW
          value: "1920"
        - name: SIZEH
          value: "1080"
        - name: REFRESH
          value: "60"
        - name: DPI
          value: "96"
        - name: CDEPTH
          value: "24"
        - name: VIDEO_PORT
          value: "DP-0"
        # Choose either `value:` or `secretKeyRef:` but not both at the same time
        - name: PASSWD
#          value: "mypasswd"
          valueFrom:
            secretKeyRef:
              name: my-pass
              key: my-pass
        # Uncomment this to enable noVNC, disabing selkies-gstreamer and ignoring all its parameters except `BASIC_AUTH_PASSWORD`, which will be used for authentication with noVNC, `BASIC_AUTH_PASSWORD` defaults to `PASSWD` if not provided
#        - name: NOVNC_ENABLE
#          value: "true"
        # Additional view-only password only applicable to the noVNC interface, choose either `value:` or `secretKeyRef:` but not both at the same time
#        - name: NOVNC_VIEWPASS
#          value: "mypasswd"
#          valueFrom:
#            secretKeyRef:
#              name: my-pass
#              key: my-pass
        ###
        # selkies-gstreamer parameters, for additional configurations see lines that start with "parser.add_argument" in https://github.com/selkies-project/selkies-gstreamer/blob/master/src/selkies_gstreamer/__main__.py
        ###
        # Change `WEBRTC_ENCODER` to `x264enc` if your GPU doesn't support `H.264 (AVCHD)` under the `NVENC - Encoding` section in https://developer.nvidia.com/video-encode-and-decode-gpu-support-matrix-new
        - name: WEBRTC_ENCODER
          value: "nvh264enc"
        - name: WEBRTC_ENABLE_RESIZE
          value: "false"
        - name: ENABLE_AUDIO
          value: "true"
        - name: ENABLE_BASIC_AUTH
          value: "true"
        # Defaults to `PASSWD` if unspecified, choose either `value:` or `secretKeyRef:` but not both at the same time
#        - name: BASIC_AUTH_PASSWORD
#          value: "mypasswd"
#          valueFrom:
#            secretKeyRef:
#              name: my-pass
#              key: my-pass
        ###
        # Uncomment below to use a TURN server for improved network compatibility
        ###
        - name: TURN_HOST
          value: "turn.nrp-nautilus.io"
        - name: TURN_PORT
          value: "3478"
        # Provide only `TURN_SHARED_SECRET` for time-limited shared secret authentication or both `TURN_USERNAME` and `TURN_PASSWORD` for legacy long-term authentication, but do not provide both authentication methods at the same time
        - name: TURN_SHARED_SECRET
          valueFrom:
            secretKeyRef:
              name: my-pass
              key: turn-secret
#        - name: TURN_USERNAME
#          value: "username"
        # Choose either `value:` or `secretKeyRef:` but not both at the same time
#        - name: TURN_PASSWORD
#          value: "mypasswd"
#          valueFrom:
#            secretKeyRef:
#              name: turn-password
#              key: turn-password
        # Change to `tcp` if the UDP protocol is throttled or blocked in your client network, or when the TURN server does not support UDP
        - name: TURN_PROTOCOL
          value: "udp"
        # You need a valid hostname and a certificate from authorities such as ZeroSSL (Let's Encrypt may have issues) to enable this
        - name: TURN_TLS
          value: "false"
        stdin: true
        tty: true
        ports:
        - name: http
          containerPort: 8080
          protocol: TCP
        resources:
          limits:
            memory: 64Gi
            cpu: "16"
            nvidia.com/gpu: 1
          requests:
            memory: 100Mi
            cpu: 100m
        volumeMounts:
        - mountPath: /dev/shm
          name: dshm
        - mountPath: /cache
          name: xgl-cache-vol
        - mountPath: /home/user
          name: xgl-root-vol
      dnsPolicy: None
      dnsConfig:
        nameservers:
        - 8.8.8.8
        - 8.8.4.4
      volumes:
      - name: dshm
        emptyDir:
          medium: Memory
      - name: xgl-cache-vol
        emptyDir: {}
#        persistentVolumeClaim:
#          claimName: xgl-cache-vol
      - name: xgl-root-vol
        emptyDir: {}
#        persistentVolumeClaim:
#          claimName: xgl-root-vol
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: nautilus.io/vis
                operator: NotIn
                values:
                - suncave
#              - key: topology.kubernetes.io/region
#                operator: In
#                values:
#                - us-west

The below is a reference configuration egl.yml for docker-nvidia-egl-desktop.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: egl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: egl
  template:
    metadata:
      labels:
        app: egl
    spec:
      hostname: egl
      containers:
      - name: egl
        image: ghcr.io/selkies-project/nvidia-egl-desktop:latest
        env:
        - name: TZ
          value: "UTC"
        # Keep to default unless you know what you are doing with VirtualGL, `VGL_DISPLAY` should be set to either `egl[n]`, or `/dev/dri/card[n]` only when the device was passed to the container
#        - name: VGL_DISPLAY
#          value: "egl"
        - name: SIZEW
          value: "1920"
        - name: SIZEH
          value: "1080"
        - name: REFRESH
          value: "60"
        - name: DPI
          value: "96"
        - name: CDEPTH
          value: "24"
        # Choose either `value:` or `secretKeyRef:` but not both at the same time
        - name: PASSWD
#          value: "mypasswd"
          valueFrom:
            secretKeyRef:
              name: my-pass
              key: my-pass
        # Uncomment this to enable noVNC, disabing selkies-gstreamer and ignoring all its parameters except `BASIC_AUTH_PASSWORD`, which will be used for authentication with noVNC, `BASIC_AUTH_PASSWORD` defaults to `PASSWD` if not provided
#        - name: NOVNC_ENABLE
#          value: "true"
        # Additional view-only password only applicable to the noVNC interface, choose either `value:` or `secretKeyRef:` but not both at the same time
#        - name: NOVNC_VIEWPASS
#          value: "mypasswd"
#          valueFrom:
#            secretKeyRef:
#              name: my-pass
#              key: my-pass
        ###
        # selkies-gstreamer parameters, for additional configurations see lines that start with "parser.add_argument" in https://github.com/selkies-project/selkies-gstreamer/blob/master/src/selkies_gstreamer/__main__.py
        ###
        # Change `WEBRTC_ENCODER` to `x264enc` if you are using software fallback without allocated GPUs or your GPU doesn't support `H.264 (AVCHD)` under the `NVENC - Encoding` section in https://developer.nvidia.com/video-encode-and-decode-gpu-support-matrix-new
        - name: WEBRTC_ENCODER
          value: "nvh264enc"
        - name: WEBRTC_ENABLE_RESIZE
          value: "false"
        - name: ENABLE_AUDIO
          value: "true"
        - name: ENABLE_BASIC_AUTH
          value: "true"
        # Defaults to `PASSWD` if unspecified, choose either `value:` or `secretKeyRef:` but not both at the same time
#        - name: BASIC_AUTH_PASSWORD
#          value: "mypasswd"
#          valueFrom:
#            secretKeyRef:
#              name: my-pass
#              key: my-pass
        ###
        # Uncomment below to use a TURN server for improved network compatibility
        ###
        - name: TURN_HOST
          value: "turn.nrp-nautilus.io"
        - name: TURN_PORT
          value: "3478"
        # Provide only `TURN_SHARED_SECRET` for time-limited shared secret authentication or both `TURN_USERNAME` and `TURN_PASSWORD` for legacy long-term authentication, but do not provide both authentication methods at the same time
        - name: TURN_SHARED_SECRET
          valueFrom:
            secretKeyRef:
              name: my-pass
              key: turn-secret
#        - name: TURN_USERNAME
#          value: "username"
        # Choose either `value:` or `secretKeyRef:` but not both at the same time
#        - name: TURN_PASSWORD
#          value: "mypasswd"
#          valueFrom:
#            secretKeyRef:
#              name: turn-password
#              key: turn-password
        # Change to `tcp` if the UDP protocol is throttled or blocked in your client network, or when the TURN server does not support UDP
        - name: TURN_PROTOCOL
          value: "udp"
        # You need a valid hostname and a certificate from authorities such as ZeroSSL (Let's Encrypt may have issues) to enable this
        - name: TURN_TLS
          value: "false"
        stdin: true
        tty: true
        ports:
        - name: http
          containerPort: 8080
          protocol: TCP
        resources:
          limits:
            memory: 64Gi
            cpu: "16"
            nvidia.com/gpu: 1
          requests:
            memory: 100Mi
            cpu: 100m
        volumeMounts:
        - mountPath: /dev/shm
          name: dshm
        - mountPath: /cache
          name: egl-cache-vol
        - mountPath: /home/user
          name: egl-root-vol
      dnsPolicy: None
      dnsConfig:
        nameservers:
        - 8.8.8.8
        - 8.8.4.4
      volumes:
      - name: dshm
        emptyDir:
          medium: Memory
      - name: egl-cache-vol
        emptyDir: {}
#        persistentVolumeClaim:
#          claimName: egl-cache-vol
      - name: egl-root-vol
        emptyDir: {}
#        persistentVolumeClaim:
#          claimName: egl-root-vol
#      affinity:
#        nodeAffinity:
#          requiredDuringSchedulingIgnoredDuringExecution:
#            nodeSelectorTerms:
#            - matchExpressions:
#              - key: topology.kubernetes.io/region
#                operator: In
#                values:
#                - us-west

Customization

In one entry, value: and valueFrom must not exist at the same time.

Comment out emptyDir: {} and uncomment persistentVolumeClaim:, then change claimName: to the name of your PersistentVolumeClaim after uncommenting.

rook-ceph-block, rook-ceph-block-east, or rook-ceph-block-pacific are the recommended StorageClasses to be mounted to /home/user. However, if performance is slow you may use rook-cephfs, rook-cephfs-east, rook-cephfs-pacific, seaweedfs-storage-hdd, or seaweedfs-storage-nvme, but NEVER MOUNT TO /home/user. Instead mount to a different directory such as /home/user/persistent or /mnt/persistent. Refer to the POSIX section for more information.

If your client network blocks (likely your campus network) or throttles (likely your home network restricted by your Internet Service Privider) the UDP protocol, change the environment variable TURN_PROTOCOL to tcp.

If your network or country enforces Deep Packet Inspection to the WebRTC or STUN/TURN protocols, set TURN_TLS to true.

Check the README.md of both GitHub repositories docker-nvidia-glx-desktop or docker-nvidia-egl-desktop for more details on configuration customization. Please give the repositories a star too!

Secret Generation

Before generating the Kubernetes secret, receive the TURN Shared Secret by asking in the Nautilus Support Matrix Chatroom, and replace TURN_SHARED_SECRET to the value that you have received. Replace YOUR_PASSWORD with the password for the container that you intend to use. If you want to use a different name instead of my-pass or turn-secret, make sure to update the above xgl.yml or egl.yml files as well as the below command. The first my-pass is the name of the secret, and the second my-pass right after --from-literal= and turn-secret are the keys.

After customizing, run the below command in edited form:

kubectl create secret generic my-pass --from-literal=my-pass=YOUR_PASSWORD --from-literal=turn-secret=TURN_SHARED_SECRET

Container Start

After saving and editing the reference configurations, run either of these commands to start the container depending on the type of your container:

kubectl create -f xgl.yml

kubectl create -f egl.yml

Exposing the Container

The below reference configuration xgl-ingress.yml is to expose your docker-nvidia-glx-desktop container to the *.nrp-nautilus.io endpoint. Replace YOUR_ENDPOINT to the subdomain you want to use. You can just use kubectl port-forward deployment/xgl 8080:8080 and access localhost:8080, but this will have higher latency and subpar performance.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: haproxy
  name: xgl
spec:
  rules:
  - host: YOUR_ENDPOINT.nrp-nautilus.io
    http:
      paths:
      - backend:
          service:
            name: xgl
            port:
              name: http
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - YOUR_ENDPOINT.nrp-nautilus.io
---
apiVersion: v1
kind: Service
metadata:
  name: xgl
  labels:
    app: xgl
spec:
  selector:
    app: xgl
  ports:
  - name: http
    protocol: TCP
    port: 8080

Run the below command after saving the changed reference configuration file: kubectl create -f xgl-ingress.yml

Access YOUR_ENDPOINT.nrp-nautilus.io with your web browser.

The below reference configuration egl-ingress.yml is to expose your docker-nvidia-egl-desktop container to the *.nrp-nautilus.io endpoint. Replace YOUR_ENDPOINT to the subdomain you want to use. You can just use kubectl port-forward deployment/egl 8080:8080 and access localhost:8080, but this will have higher latency and subpar performance.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: haproxy
  name: egl
spec:
  rules:
  - host: YOUR_ENDPOINT.nrp-nautilus.io
    http:
      paths:
      - backend:
          service:
            name: egl
            port:
              name: http
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - YOUR_ENDPOINT.nrp-nautilus.io
---
apiVersion: v1
kind: Service
metadata:
  name: egl
  labels:
    app: egl
spec:
  selector:
    app: egl
  ports:
  - name: http
    protocol: TCP
    port: 8080

Run the below command after saving the changed reference configuration file: kubectl create -f egl-ingress.yml

Access YOUR_ENDPOINT.nrp-nautilus.io with your web browser. The user name is "user" without the quotes and the password is the my-pass secret that you have set.

Container Customization

You can import either https://github.com/selkies-project/docker-nvidia-glx-desktop.git or https://github.com/selkies-project/docker-nvidia-egl-desktop.git by importing with Repo by URL from https://gitlab.nrp-nautilus.io/projects/new#import_project. Refer to Building in GitLab on how you can change and build your own customized container.

If you are changing the Dockerfile, you are recommended to use the containers as a base container and only replace the entrypoint.sh and supervisord.conf files. This will keep you up to date with the latest updates.

To do this, start with these sample Dockerfile examples and place your modified entrypoint.sh and supervisord.conf files within the same git directory:

ARG UBUNTU_RELEASE=22.04
FROM ghcr.io/selkies-project/nvidia-glx-desktop:${UBUNTU_RELEASE}
ARG UBUNTU_RELEASE

USER root
# Install and configure below this line

# Replace changed files
#COPY entrypoint.sh /etc/entrypoint.sh
#RUN chmod 755 /etc/entrypoint.sh
#COPY supervisord.conf /etc/supervisord.conf
#RUN chmod 755 /etc/supervisord.conf

EXPOSE 8080

USER user
ENV USER=user
WORKDIR /home/user

ENTRYPOINT ["/usr/bin/supervisord"]

or

ARG UBUNTU_RELEASE=22.04
FROM ghcr.io/selkies-project/nvidia-egl-desktop:${UBUNTU_RELEASE}
ARG UBUNTU_RELEASE

USER root
# Install and configure below this line

# Replace changed files
#COPY entrypoint.sh /etc/entrypoint.sh
#RUN chmod 755 /etc/entrypoint.sh
#COPY supervisord.conf /etc/supervisord.conf
#RUN chmod 755 /etc/supervisord.conf

EXPOSE 8080

USER user
ENV USER=user
WORKDIR /home/user

ENTRYPOINT ["/usr/bin/supervisord"]

Please use the below .gitlab-ci.yml file to build the container on GitLab and push to the container repository (set UBUNTU_RELEASE to any supported Ubuntu LTS release):

image: gcr.io/kaniko-project/executor:debug
stages:
  - build-and-push
build-and-push-job:
  stage: build-and-push
  script:
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
    - /kaniko/executor --use-new-run=true --snapshot-mode=redo --cache=true --push-retry=10 --build-arg=UBUNTU_RELEASE=22.04 --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $CI_REGISTRY_IMAGE:${CI_COMMIT_SHA:0:8} --destination $CI_REGISTRY_IMAGE:latest

Again, please give both docker-nvidia-glx-desktop and docker-nvidia-egl-desktop a star if the containers were useful to you.