mw-pfeddersheim-workstation/docs/tech/crash-analysis-2026-03-17.md

Crash Analysis Report - 2026-03-17

Incident Overview

• Timestamp: 2026-03-17 18:33 (approx. 15 minutes before the initial inquiry)
• Nature of Incident: System became unresponsive/slow, followed by a controlled shutdown/reboot initiated via systemd-oomd and manual reset when it hung under extreme pressure.
• Root Cause: Severe memory pressure (RAM + Swap both > 90%).

Detailed Findings

1. Memory Depletion & Process Behavior

At the time of the crash (18:32:36):

• RAM: ~61.8GiB used (66,387,353,600 bytes) / 62.7GiB total (98.6%).
• Swap: ~3.6GiB used (3,889,168,384 bytes) / 4GiB total (90.5%).
• Key Culprit: The unit kitty-4851-1.scope (Kitty terminal) was the primary consumer, reaching a 9.4G memory peak and 1.2G swap peak.
• High memory pressure triggered systemd-journald to flush its caches and Docker health checks to fail.

2. OOM Intervention (systemd-oomd)

systemd-oomd (user-space OOM killer) correctly identified the pressure on /user.slice at 18:32:36 and terminated processes in the following unit:

• Unit: /user.slice/user-1000.slice/user@1000.service/app.slice/kitty-4851-1.scope
• Result: 16 processes were killed within this scope.

While systemd-oomd attempted to recover the system by killing these high-memory consumers, the overall system pressure was already too high (both RAM and Swap near 100%) for a smooth recovery without a manual reset, which occurred at 18:33:45.

3. Current System State (Post-Reboot)

The system is now running with the following configuration:

• Kernel: 6.12.73-1-MANJARO
• Swap: 16GB file (/swapfile).
• zswap: Active (zstd/zsmalloc, 20% max pool). This was previously "pending" and is now successfully loaded.
• systemd-oomd: Running and monitoring /user.slice with 90% swap and 60%/30s memory pressure limits.
• Hardware: NVMe disk self-assessment (S.M.A.R.T.) result: PASSED.

Recommendations & Optimizations

1. Increase Swap Size (Completed)

The swap file has been increased from 4GB to 16GB via Ansible (2026-03-17). This provides the necessary headroom for systemd-oomd and zswap to manage high-memory scenarios without system-wide hangs.

2. Tune zswap for High-RAM Systems (Completed)

The current 30% max_pool_percent for zswap (approx. 19.2GB) is configured via GRUB and active at runtime. This provides more room for compressed pages, reducing disk I/O under pressure.

3. Kernel Memory Optimizations (Completed)

Several kernel parameters were tuned via Ansible (2026-03-17) to improve interactive response and proactive memory reclamation:

• Transparent Hugepages (THP): Set to madvise to prevent allocation stalls while still allowing performance gains for optimized apps.
• Proactive Reclaim: vm.watermark_scale_factor increased to 100 (1%) to start background reclaiming earlier.
• Dirty Page Management: vm.dirty_ratio and vm.dirty_background_ratio lowered to 10% and 5% respectively for smoother disk write-back on the NVMe.
• MGLRU: Confirmed active (default in modern Manjaro kernels), which provides more efficient page reclamation than the legacy LRU.

4. Consider EarlyOOM (Alternative)

While systemd-oomd is the current standard, earlyoom can sometimes be more responsive for desktop users who need immediate termination of runaway processes before the system hangs.

• Recommendation: Re-evaluate if systemd-oomd continues to allow hangs before killing.

Conclusion

The recent "crash" was a classic Out-Of-Memory scenario where the newly installed systemd-oomd successfully intervened, but the lack of swap headroom led to a total system freeze. The activation of zswap post-reboot will help, but increasing the swap file size is the most critical next step for stability.