Proxmox freeze without a log entry: troubleshooting a ThinkCentre M920q
In October I bought a used Lenovo ThinkCentre M920q for my homelab with Home Assistant and a few small services (i5-9500T, 16 GB RAM, 256 GB NVMe): reasonably cheap, built to run around the clock, and small. Before putting it into service I dutifully ran memtest86+, then installed Proxmox VE 9 with a Home Assistant VM and a few LXC containers. Shortly after, it started freezing again and again, five times in the first three days alone. Each time it became unreachable, and the journal had nothing that would have explained it. The story only has half a happy ending: I never found the cause, but an identical replacement has been running without problems ever since.
I'm not an infrastructure pro. I used to avoid the topic (but hey, these days I'm writing a series about NixOS). So during the troubleshooting I had to look up a lot of things with Kagi, and what I found and tried along the way is all here.
journalctl, my new best friend
The first freeze happened on October 8, and only a reset on the device helped. Afterwards I wanted to know what had happened before, and that's how I got to know journalctl. The kernel messages that dmesg shows are gone after a reset. The systemd journal, on the other hand, keeps the messages from earlier boots as well, and those were exactly what I needed. My Proxmox installation did that out of the box. If journalctl --list-boots shows only one entry, the journal lives in memory. In that case, create the directory /var/log/journal and restart the service with systemctl restart systemd-journald.
journalctl -b -1 -e shows the end of the previous boot (-b -1 is the boot before and -e jumps to the end). In my case the log simply stopped in the middle of normal operation, without a warning or an error before it.
The next freezes came sometimes after an hour, sometimes after 35 hours, with no recognizable pattern.
The usual suspects
The first thing I did was update the BIOS, which unfortunately didn't help. Then I took on two errors from the kernel log, one from the network card and one from the NVMe, and for both I quickly found threads with the same advice over and over. On October 7, the network card (e1000e) had reported a "Hardware Unit Hang":
journalctl -k | grep -i "hardware unit hang"The usual fix is to turn off offloading, where the network card takes work like splitting up large packets off the CPU. To make it permanent, it goes into /etc/network/interfaces, on the physical card (ip link shows its name, on the M920q usually eno1), not on the bridge vmbr0 that Proxmox attaches the VMs to. post-up runs the command every time the card comes up:
iface eno1 inet manual
post-up /sbin/ethtool -K eno1 \
tso off gso off gro offShortly before the first freeze, there was also a PCIe error from the NVMe in the log (AER, RxErr, found with journalctl -k | grep -i aer). That's a corrected error the hardware has already fixed by itself, and on its own it's often harmless. Still, the advice in the threads was to turn off ASPM, the power saving feature of the PCIe link. That works with the kernel parameter pcie_aspm=off, an option the bootloader passes to the kernel at startup. Depending on the installation, Proxmox uses GRUB or systemd-boot, and proxmox-boot-tool status tells you which. With GRUB, you append the parameter to this line in /etc/default/grub:
GRUB_CMDLINE_LINUX_DEFAULT="quiet pcie_aspm=off"Then run update-grub and reboot. cat /proc/cmdline shows whether the parameter is active. With systemd-boot, the parameters live in /etc/kernel/cmdline instead, and the command is proxmox-boot-tool refresh. The details are in the Proxmox docs on the bootloader.
At some point I noticed that the case was warm. At idle, the CPU ran at 3.5 GHz and 43 to 59 °C, because the governor, which controls the clock speed, was set to performance. It can be changed per core under /sys. This command does it for all of them at once, until the next reboot:
echo powersave | tee \
/sys/devices/system/cpu/cpu*/cpufreq/scaling_governorWith offloading and ASPM off, both error messages were gone, and with powersave the CPU was noticeably cooler at idle, at 34 to 36 °C. The host kept freezing anyway. Whether the two errors had anything to do with the freezes at all, I still don't know.
A temperature log and a false suspicion
I wanted to know more precisely whether heat was the problem after all, especially how warm the machine was in the minutes before a freeze. So I wrote a small script that reads the highest temperature of all sensors under /sys/class/thermal/ (the kernel reports it in thousandths of a degree), plus the average clock speed from /proc/cpuinfo and the load of the last minute from /proc/loadavg. logger -t templog writes that as a single line to the journal, where it survives the freeze. On my machine the script lives at /usr/local/bin/templog.sh:
#!/bin/sh
t=$(cat /sys/class/thermal/thermal_zone*/temp \
| sort -n | tail -1)
f=$(awk '/MHz/ {s+=$4; n++} \
END {printf "%.0f", s/n}' /proc/cpuinfo)
l=$(cut -d ' ' -f 1 /proc/loadavg)
logger -t templog "temp=$((t/1000))C freq=${f}MHz load=$l"chmod +x makes it executable, and after a test run by hand, journalctl -t templog -n 1 shows whether the line arrives. To run it regularly, it goes into root's crontab. crontab -e opens it in an editor. To start with, I had it log every minute:
* * * * * /usr/local/bin/templog.shThe line in the journal then looks roughly like this: temp=36C freq=800MHz load=0.12. After a freeze, journalctl -b -1 -t templog | tail shows the last few minutes. During the next freezes, the temperature stayed at 34 to 36 °C right up to the end, so heat wasn't the problem.
If you stare at logs long enough, you start seeing patterns. I got there when I noticed that three of the first four freezes ended in exactly the second a container sent a heartbeat to my Uptime Kuma (post in German). That couldn't be a coincidence. So I moved the heartbeat by two minutes to see whether the freeze would move with it. At the next freeze it didn't, so it was a coincidence after all. Once you start suspecting your own monitoring, it might be time for a break.
The lockup panic
At some point I turned the search around and asked how I could get an error message out of a freeze at all. That's how I came across hardlockup_panic and softlockup_panic. The kernel uses a watchdog to check regularly whether every CPU core still responds. If one is stuck, that's a lockup, and normally all you get is a warning in the log. With these two settings, the kernel triggers a panic instead and saves the message.
Kernel settings like these are called sysctl, and to make them permanent they go into a file under /etc/sysctl.d/, mine is 99-lockup.conf. kernel.panic = 10 reboots the machine ten seconds after the panic instead of leaving it hanging:
kernel.hardlockup_panic = 1
kernel.softlockup_panic = 1
kernel.panic = 10sysctl --system loads the file right away, and sysctl kernel.hardlockup_panic shows whether it worked.
Where it saves the message is the clever part. During a panic, the kernel can no longer write to disk, so the journal gets nothing. But it can still write to the pstore, which on UEFI machines is usually a small storage area in the firmware, and whatever ends up there survives the reboot. Afterwards, systemd-pstore fetches the messages from the firmware and puts them under /var/lib/systemd/pstore/, one folder per panic with the last kernel messages before the crash.
You can test whether the whole chain works. The first command shows whether the NMI watchdog is running, without which the kernel can't detect a hard lockup. The second one triggers a panic on purpose. That's a hard crash, so shut down all VMs and containers first. With kernel.panic = 10 the machine reboots after ten seconds, without that setting it just stays down:
cat /proc/sys/kernel/nmi_watchdog # should be 1
echo c > /proc/sysrq-triggerIf there's a folder under /var/lib/systemd/pstore/ afterwards, the pstore works.
At the next freeze, the machine didn't reboot, and nothing ended up in the pstore either. That was the most revealing failure of the whole series, even though it took me some reading to understand why. The missing reboot is what tells you the most, because I don't know whether the pstore was even active on my machine. The watchdog never noticed the standstill. If it was running, it wasn't just one core that hung, but the whole machine. That points to the hardware rather than to a container doing something stupid. Until then, I had assumed without noticing that the kernel was alive and just doing something wrong.
C-states: the last software idea
I did find one more explanation without a hardware defect. The Proxmox forum has several reports of hosts that freeze exactly like this, without a log entry. The suspicion there: a CPU core gets stuck in a deep sleep state (C-state) and never wakes up again. The higher the number, the deeper the core sleeps and the more power it saves. On Intel CPUs, the intel_idle driver handles this, and you can see which states it uses on your machine with:
cat /sys/devices/system/cpu/cpu0/cpuidle/state*/nameMine went up to C10. The parameter intel_idle.max_cstate=1 only allows C1, the lightest sleep. It's added the same way as pcie_aspm=off before:
GRUB_CMDLINE_LINUX_DEFAULT="quiet pcie_aspm=off intel_idle.max_cstate=1"cat /sys/module/intel_idle/parameters/max_cstate shows whether it took effect (output: 1). To be safe, I also disabled C-states in the BIOS. I had high hopes that this would save me from sending the machine back. It didn't help, the host still froze, just as silently as before. A different power supply didn't change anything either.
The replacement
The machine was still within the return period, so it went back. The replacement is an identical M920q with the same Proxmox installation and the same guests, but without pcie_aspm=off, without max_cstate=1, with the default governor and offloading turned on, and it has run for a month without a freeze.
With the old unit, there were at most 35 hours between the first freezes, so a month without one is a pretty clear result. It was this particular unit. Where exactly, I don't know, but it wasn't the power supply. My guess is the mainboard.
What I would check first today
Not everyone can return a device. Before you buy a replacement, I would work through this list:
- Run
memtest86+, always first, ideally before putting the machine into service. Proxmox already has it in the boot menu. It takes a while. A long while. A very long while. It's worth it. - Update the BIOS and install the
intel-microcodepackage, which brings updates for the CPU. It's in thenon-free-firmwarerepository, which has to be enabled for that. Proxmox recommends both. Whether my microcode was up to date, I no longer know. - Enable the lockup panic right after the first freeze, and check with the panic test above that it works. If everything still stays silent during the next freeze, I would only try the C-states and then suspect the hardware.
- Swap the power supply. It didn't change anything for me, but for mini PCs with an external power brick it's the cheapest test.
- Disable C-states in the BIOS instead of only limiting them with a kernel parameter. That didn't help me either, but it's quick to try.
To be honest, I suspected the hardware from the start. I still tried everything I could find, because it's also kind of exciting, and I'd do it the same way again. I'd just do the simple hardware tests earlier.
My journalctl cheat sheet
By now I know journalctl a lot better. I wrote these commands down for my future self:
# all boots with first and last entry
journalctl --list-boots
# end of the previous boot
journalctl -b -1 -e
# kernel messages only
journalctl -b -1 -k
# only errors and worse
journalctl -b -1 -p err
# narrow down a time range
journalctl --since "2026-10-08 10:00" --until "2026-10-08 11:00"
# search for text, no grep needed
journalctl -k -g "hardware unit hang"
# one service, e.g. the Proxmox web UI
journalctl -u pveproxy
# only entries with your own tag (logger -t)
journalctl -t templog
# follow live
journalctl -f