mdcTUX
June 7, 2023, 10:37am
1
When I try to start an container rootless, it will fails.
sudo -u <user> podman container start jovial_poitras
Error: OCI runtime error: unable to start container "b1d4a290d9219374263aff7727fd04789e859b10d034a63bdc8992ae12e13bbb": crun: setgroups: Invalid argument
The download of the image (centos:8) was successful.
sudo -u <user> podman unshare cat /proc/self/gid_map
0 71426 1
1 110000 65536
sudo -u j<user> podman unshare cat /proc/self/uid_map
0 71426 1
1 110000 65536
On Rocky 8.8 the same was working.
No it still works, I am running all my containers rootless on Rocky Linux 9. Did you install “crun”?
mdcTUX
June 7, 2023, 11:52am
3
Yes, crun is installed.
rpm -qf /usr/bin/crun
crun-1.8.4-1.el9_2.x86_64
The only difference I think between how you are doing it and how I am doing is, is that I make an ssh connection to the user running a container. What happens when you ssh to that user and try to start the container instead of using sudo?
mdcTUX
June 7, 2023, 11:59am
5
This will result in the same
I would check to see if you see any useful log information in messages when the container you are trying to start fails to start and maybe also check if the failed containers contains any useful log information.
mdcTUX
June 8, 2023, 11:36am
7
Not really, but here an debug output for it:
sudo -u <user> podman --log-level debug container start sad_heisenberg >/tmp/debug.log 2>&1
will result in:
time="2023-06-08T13:32:38+02:00" level=info msg="podman filtering at log level debug"
time="2023-06-08T13:32:38+02:00" level=debug msg="Called start.PersistentPreRunE(podman --log-level debug container start sad_heisenberg)"
time="2023-06-08T13:32:38+02:00" level=debug msg="Using conmon: \"/usr/bin/conmon\""
time="2023-06-08T13:32:38+02:00" level=debug msg="Initializing boltdb state at /var/tmp/jenkins/.local/share/containers/storage/libpod/bolt_state.db"
time="2023-06-08T13:32:38+02:00" level=debug msg="Overriding run root \"/tmp/containers-user-71426/containers\" with \"/run/user/71426/containers\" from database"
time="2023-06-08T13:32:38+02:00" level=debug msg="systemd-logind: Unknown object '/'."
time="2023-06-08T13:32:38+02:00" level=debug msg="Using graph driver overlay"
time="2023-06-08T13:32:38+02:00" level=debug msg="Using graph root /var/tmp/jenkins/.local/share/containers/storage"
time="2023-06-08T13:32:38+02:00" level=debug msg="Using run root /run/user/71426/containers"
time="2023-06-08T13:32:38+02:00" level=debug msg="Using static dir /var/tmp/jenkins/.local/share/containers/storage/libpod"
time="2023-06-08T13:32:38+02:00" level=debug msg="Using tmp dir /run/user/71426/libpod/tmp"
time="2023-06-08T13:32:38+02:00" level=debug msg="Using volume path /var/tmp/jenkins/.local/share/containers/storage/volumes"
time="2023-06-08T13:32:38+02:00" level=debug msg="Using transient store: false"
time="2023-06-08T13:32:38+02:00" level=debug msg="Set libpod namespace to \"\""
time="2023-06-08T13:32:38+02:00" level=debug msg="Not configuring container store"
time="2023-06-08T13:32:38+02:00" level=debug msg="Initializing event backend journald"
time="2023-06-08T13:32:38+02:00" level=debug msg="Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument"
time="2023-06-08T13:32:38+02:00" level=debug msg="Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument"
time="2023-06-08T13:32:38+02:00" level=debug msg="Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument"
time="2023-06-08T13:32:38+02:00" level=debug msg="Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument"
time="2023-06-08T13:32:38+02:00" level=debug msg="Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument"
time="2023-06-08T13:32:38+02:00" level=debug msg="Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument"
time="2023-06-08T13:32:38+02:00" level=debug msg="Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument"
time="2023-06-08T13:32:38+02:00" level=debug msg="Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument"
time="2023-06-08T13:32:38+02:00" level=debug msg="Using OCI runtime \"/usr/bin/crun\""
time="2023-06-08T13:32:38+02:00" level=info msg="Setting parallel job count to 61"
time="2023-06-08T13:32:38+02:00" level=info msg="podman filtering at log level debug"
time="2023-06-08T13:32:38+02:00" level=debug msg="Called start.PersistentPreRunE(podman --log-level debug container start sad_heisenberg)"
time="2023-06-08T13:32:38+02:00" level=debug msg="Using conmon: \"/usr/bin/conmon\""
time="2023-06-08T13:32:38+02:00" level=debug msg="Initializing boltdb state at /var/tmp/jenkins/.local/share/containers/storage/libpod/bolt_state.db"
time="2023-06-08T13:32:38+02:00" level=debug msg="systemd-logind: Unknown object '/'."
time="2023-06-08T13:32:38+02:00" level=debug msg="Using graph driver overlay"
time="2023-06-08T13:32:38+02:00" level=debug msg="Using graph root /var/tmp/jenkins/.local/share/containers/storage"
time="2023-06-08T13:32:38+02:00" level=debug msg="Using run root /run/user/71426/containers"
time="2023-06-08T13:32:38+02:00" level=debug msg="Using static dir /var/tmp/jenkins/.local/share/containers/storage/libpod"
time="2023-06-08T13:32:38+02:00" level=debug msg="Using tmp dir /run/user/71426/libpod/tmp"
time="2023-06-08T13:32:38+02:00" level=debug msg="Using volume path /var/tmp/jenkins/.local/share/containers/storage/volumes"
time="2023-06-08T13:32:38+02:00" level=debug msg="Using transient store: false"
time="2023-06-08T13:32:38+02:00" level=debug msg="Set libpod namespace to \"\""
time="2023-06-08T13:32:38+02:00" level=debug msg="[graphdriver] trying provided driver \"overlay\""
time="2023-06-08T13:32:38+02:00" level=debug msg="Cached value indicated that overlay is supported"
time="2023-06-08T13:32:38+02:00" level=debug msg="Cached value indicated that overlay is supported"
time="2023-06-08T13:32:38+02:00" level=debug msg="Cached value indicated that metacopy is not being used"
time="2023-06-08T13:32:38+02:00" level=debug msg="Cached value indicated that native-diff is usable"
time="2023-06-08T13:32:38+02:00" level=debug msg="backingFs=xfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false"
time="2023-06-08T13:32:38+02:00" level=debug msg="Initializing event backend journald"
time="2023-06-08T13:32:38+02:00" level=debug msg="Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument"
time="2023-06-08T13:32:38+02:00" level=debug msg="Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument"
time="2023-06-08T13:32:38+02:00" level=debug msg="Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument"
time="2023-06-08T13:32:38+02:00" level=debug msg="Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument"
time="2023-06-08T13:32:38+02:00" level=debug msg="Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument"
time="2023-06-08T13:32:38+02:00" level=debug msg="Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument"
time="2023-06-08T13:32:38+02:00" level=debug msg="Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument"
time="2023-06-08T13:32:38+02:00" level=debug msg="Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument"
time="2023-06-08T13:32:38+02:00" level=debug msg="Using OCI runtime \"/usr/bin/crun\""
time="2023-06-08T13:32:38+02:00" level=info msg="Setting parallel job count to 61"
time="2023-06-08T13:32:38+02:00" level=debug msg="Made network namespace at /run/user/71426/netns/netns-863a9764-f822-f2a1-7ed2-2aee807ec549 for container d525de0149549fd6f2e0a039de2ce3ac621e1a339b9e660c9c921f71cc2e46a6"
time="2023-06-08T13:32:38+02:00" level=debug msg="Cached value indicated that idmapped mounts for overlay are not supported"
time="2023-06-08T13:32:38+02:00" level=debug msg="Check for idmapped mounts support "
time="2023-06-08T13:32:38+02:00" level=debug msg="overlay: mount_data=lowerdir=/var/tmp/jenkins/.local/share/containers/storage/overlay/l/34FNF6ELSNOJ4VWNE6GDL55TLV,upperdir=/var/tmp/jenkins/.local/share/containers/storage/overlay/6080b02fada2d261093b58d0423e4f602c7dee5a957f6fa9d6f84dbc502c21e3/diff,workdir=/var/tmp/jenkins/.local/share/containers/storage/overlay/6080b02fada2d261093b58d0423e4f602c7dee5a957f6fa9d6f84dbc502c21e3/work,,userxattr,context=\"system_u:object_r:container_file_t:s0:c417,c470\""
time="2023-06-08T13:32:38+02:00" level=debug msg="slirp4netns command: /bin/slirp4netns --disable-host-loopback --mtu=65520 --enable-sandbox --enable-seccomp --enable-ipv6 -c -e 3 -r 4 --netns-type=path /run/user/71426/netns/netns-863a9764-f822-f2a1-7ed2-2aee807ec549 tap0"
time="2023-06-08T13:32:38+02:00" level=debug msg="Mounted container \"d525de0149549fd6f2e0a039de2ce3ac621e1a339b9e660c9c921f71cc2e46a6\" at \"/var/tmp/jenkins/.local/share/containers/storage/overlay/6080b02fada2d261093b58d0423e4f602c7dee5a957f6fa9d6f84dbc502c21e3/merged\""
time="2023-06-08T13:32:38+02:00" level=debug msg="Created root filesystem for container d525de0149549fd6f2e0a039de2ce3ac621e1a339b9e660c9c921f71cc2e46a6 at /var/tmp/jenkins/.local/share/containers/storage/overlay/6080b02fada2d261093b58d0423e4f602c7dee5a957f6fa9d6f84dbc502c21e3/merged"
time="2023-06-08T13:32:38+02:00" level=debug msg="Not modifying container d525de0149549fd6f2e0a039de2ce3ac621e1a339b9e660c9c921f71cc2e46a6 /etc/passwd"
time="2023-06-08T13:32:38+02:00" level=debug msg="Not modifying container d525de0149549fd6f2e0a039de2ce3ac621e1a339b9e660c9c921f71cc2e46a6 /etc/group"
time="2023-06-08T13:32:38+02:00" level=debug msg="/etc/system-fips does not exist on host, not mounting FIPS mode subscription"
time="2023-06-08T13:32:38+02:00" level=debug msg="Setting Cgroups for container d525de0149549fd6f2e0a039de2ce3ac621e1a339b9e660c9c921f71cc2e46a6 to user.slice:libpod:d525de0149549fd6f2e0a039de2ce3ac621e1a339b9e660c9c921f71cc2e46a6"
time="2023-06-08T13:32:38+02:00" level=debug msg="reading hooks from /usr/share/containers/oci/hooks.d"
time="2023-06-08T13:32:38+02:00" level=debug msg="Workdir \"/var/tmp/jenkins/workspace/grant_pgsql_master\" resolved to a volume or mount"
time="2023-06-08T13:32:38+02:00" level=debug msg="Created OCI spec for container d525de0149549fd6f2e0a039de2ce3ac621e1a339b9e660c9c921f71cc2e46a6 at /var/tmp/jenkins/.local/share/containers/storage/overlay-containers/d525de0149549fd6f2e0a039de2ce3ac621e1a339b9e660c9c921f71cc2e46a6/userdata/config.json"
time="2023-06-08T13:32:38+02:00" level=debug msg="/usr/bin/conmon messages will be logged to syslog"
time="2023-06-08T13:32:38+02:00" level=debug msg="running conmon: /usr/bin/conmon" args="[--api-version 1 -c d525de0149549fd6f2e0a039de2ce3ac621e1a339b9e660c9c921f71cc2e46a6 -u d525de0149549fd6f2e0a039de2ce3ac621e1a339b9e660c9c921f71cc2e46a6 -r /usr/bin/crun -b /var/tmp/jenkins/.local/share/containers/storage/overlay-containers/d525de0149549fd6f2e0a039de2ce3ac621e1a339b9e660c9c921f71cc2e46a6/userdata -p /run/user/71426/containers/overlay-containers/d525de0149549fd6f2e0a039de2ce3ac621e1a339b9e660c9c921f71cc2e46a6/userdata/pidfile -n sad_heisenberg --exit-dir /run/user/71426/libpod/tmp/exits --full-attach -s -l journald --log-level debug --syslog -t --conmon-pidfile /run/user/71426/containers/overlay-containers/d525de0149549fd6f2e0a039de2ce3ac621e1a339b9e660c9c921f71cc2e46a6/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/tmp/jenkins/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/71426/containers --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/71426/libpod/tmp --exit-command-arg --network-config-dir --exit-command-arg --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /var/tmp/jenkins/.local/share/containers/storage/volumes --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg d525de0149549fd6f2e0a039de2ce3ac621e1a339b9e660c9c921f71cc2e46a6]"
time="2023-06-08T13:32:38+02:00" level=info msg="Running conmon under slice user.slice and unitName libpod-conmon-d525de0149549fd6f2e0a039de2ce3ac621e1a339b9e660c9c921f71cc2e46a6.scope"
time="2023-06-08T13:32:38+02:00" level=debug msg="Received: -1"
time="2023-06-08T13:32:38+02:00" level=debug msg="Cleaning up container d525de0149549fd6f2e0a039de2ce3ac621e1a339b9e660c9c921f71cc2e46a6"
time="2023-06-08T13:32:38+02:00" level=debug msg="Tearing down network namespace at /run/user/71426/netns/netns-863a9764-f822-f2a1-7ed2-2aee807ec549 for container d525de0149549fd6f2e0a039de2ce3ac621e1a339b9e660c9c921f71cc2e46a6"
time="2023-06-08T13:32:38+02:00" level=debug msg="Unmounted container \"d525de0149549fd6f2e0a039de2ce3ac621e1a339b9e660c9c921f71cc2e46a6\""
Error: OCI runtime error: unable to start container "d525de0149549fd6f2e0a039de2ce3ac621e1a339b9e660c9c921f71cc2e46a6": crun: setgroups: Invalid argument
time="2023-06-08T13:32:38+02:00" level=debug msg="Shutting down engines"
