Debugging 90s hangs during shutdown on Ubuntu 20.04
I spent some time yesterday trying to understand why one of the machines here seemed to always take 90s before shutting down or rebooting. This article is a summary of the wild goose chase I embarked on to try and understand what was happening.
When systemd machines hang during shutdown, it usually means that there's a service that is not stopping fast enough
or it could be that systemd is waiting for the DefaultTimeoutStopSec timeout before
killing the remaining processes after all services have been stopped.
In my case, what I was seeing is the following:
[ OK ] Finished Reboot.
[ OK ] Reached target Reboot.
[ 243.652848 ] systemd-shutdown[1]: Waiting for process: containerd-shim, containerd-shim, containerd-shim, fluent-bit
These processes are part or k3s, they are the parent processes of containers or processes running
in the containers themselves. Clearly, it has something to do with k3s.
Update: the systemd bugfix was backported to Ubuntu 20.04.