docs(system): analysis reports and EOD workflows

- Created crash analysis for 2026-03-17 OOM event. - Documented stability test results and memory tuning effects. - Added PARA-structured workflows for EOD protocol. - Updated workstation health report for current 16GB swap status.
2026-03-17 19:02:06 +01:00
parent 40d073f0dc
commit 54ab23dce9
9 changed files with 198 additions and 9 deletions
@@ -0,0 +1,41 @@
+# Stability Tests Report - 2026-03-17
+
+## Executive Summary
+Following the system hang incident on 2026-03-17 at 17:44 (characterized by movable mouse but no input), several stability measures were implemented and tested. The system is now protected by a proactive user-space OOM killer (`systemd-oomd`) and strictly enforced Docker resource limits.
+
+## Implemented Measures
+
+### 1. Proactive Memory Management (`systemd-oomd`)
+- **Configuration**: Enabled `systemd-oomd` and configured specific monitoring for `/user.slice`.
+- **Thresholds**:
+  - **Memory Pressure**: Kills processes if memory pressure exceeds 60% for more than 30 seconds.
+  - **Swap Usage**: Monitored at the `/user.slice` level (90% limit).
+- **Verification**: Confirmed via `oomctl` that `/user.slice` is actively monitored with the specified thresholds.
+
+### 2. Docker Container Hardening
+- **Service: Firebird**
+  - **Memory Limit**: 4GiB (Hard limit enforced via `deploy.resources.limits.memory`).
+  - **Current Status**: Running and verified with `docker stats`.
+- **Service: Mailpit**
+  - **Memory Limit**: 512MiB (Hard limit enforced via `deploy.resources.limits.memory`).
+  - **Current Status**: Running and verified with `docker stats`.
+
+### 3. Swap Efficiency (`zswap`)
+- **Status**: Configured in GRUB (`zswap.enabled=1`).
+- **Pending**: A system reboot is required to activate the compressed swap cache.
+- **Current Status**: `N` (Disabled until next boot).
+
+## Verification & Stress Testing
+
+### Stress Test Results
+- **Methodology**: A Python script was used to rapidly allocate memory in 512MiB increments within the user slice.
+- **Observations**:
+  - The system tracked memory growth accurately in `oomctl`.
+  - Memory pressure statistics remained within safe bounds during controlled growth.
+  - No system-wide hangs occurred during rapid allocation of ~13.5GiB.
+- **Outcome**: The monitoring infrastructure is active and reporting correct metrics.
+
+## Conclusion
+The system is significantly more resilient to the "memory thrashing" scenario that caused the recent crash. The combination of `systemd-oomd` and Docker limits prevents any single user process or container from consuming all system resources and blocking kernel execution.
+
+**Action Required**: Schedule a system reboot at the earliest convenience to enable `zswap` for improved swap performance under load.