infrastructure/mw-pfeddersheim-workstation

Files

T

ja 54ab23dce9 docs(system): analysis reports and EOD workflows

- Created crash analysis for 2026-03-17 OOM event.
- Documented stability test results and memory tuning effects.
- Added PARA-structured workflows for EOD protocol.
- Updated workstation health report for current 16GB swap status.

2026-03-17 19:02:06 +01:00

2.3 KiB

Raw Permalink Blame History

Stability Tests Report - 2026-03-17

Executive Summary

Following the system hang incident on 2026-03-17 at 17:44 (characterized by movable mouse but no input), several stability measures were implemented and tested. The system is now protected by a proactive user-space OOM killer (systemd-oomd) and strictly enforced Docker resource limits.

Implemented Measures

1. Proactive Memory Management (`systemd-oomd`)

Configuration: Enabled systemd-oomd and configured specific monitoring for /user.slice.
Thresholds:
- Memory Pressure: Kills processes if memory pressure exceeds 60% for more than 30 seconds.
- Swap Usage: Monitored at the /user.slice level (90% limit).
Verification: Confirmed via oomctl that /user.slice is actively monitored with the specified thresholds.

2. Docker Container Hardening

Service: Firebird
- Memory Limit: 4GiB (Hard limit enforced via deploy.resources.limits.memory).
- Current Status: Running and verified with docker stats.
Service: Mailpit
- Memory Limit: 512MiB (Hard limit enforced via deploy.resources.limits.memory).
- Current Status: Running and verified with docker stats.

3. Swap Efficiency (`zswap`)

Status: Configured in GRUB (zswap.enabled=1).
Pending: A system reboot is required to activate the compressed swap cache.
Current Status: N (Disabled until next boot).

Verification & Stress Testing

Stress Test Results

Methodology: A Python script was used to rapidly allocate memory in 512MiB increments within the user slice.
Observations:
- The system tracked memory growth accurately in oomctl.
- Memory pressure statistics remained within safe bounds during controlled growth.
- No system-wide hangs occurred during rapid allocation of ~13.5GiB.
Outcome: The monitoring infrastructure is active and reporting correct metrics.

Conclusion

The system is significantly more resilient to the "memory thrashing" scenario that caused the recent crash. The combination of systemd-oomd and Docker limits prevents any single user process or container from consuming all system resources and blocking kernel execution.

Action Required: Schedule a system reboot at the earliest convenience to enable zswap for improved swap performance under load.

2.3 KiB Raw Permalink Blame History