ja
54ab23dce9
- Created crash analysis for 2026-03-17 OOM event. - Documented stability test results and memory tuning effects. - Added PARA-structured workflows for EOD protocol. - Updated workstation health report for current 16GB swap status.
42 lines
2.3 KiB
Markdown
42 lines
2.3 KiB
Markdown
# Stability Tests Report - 2026-03-17
|
|
|
|
## Executive Summary
|
|
Following the system hang incident on 2026-03-17 at 17:44 (characterized by movable mouse but no input), several stability measures were implemented and tested. The system is now protected by a proactive user-space OOM killer (`systemd-oomd`) and strictly enforced Docker resource limits.
|
|
|
|
## Implemented Measures
|
|
|
|
### 1. Proactive Memory Management (`systemd-oomd`)
|
|
- **Configuration**: Enabled `systemd-oomd` and configured specific monitoring for `/user.slice`.
|
|
- **Thresholds**:
|
|
- **Memory Pressure**: Kills processes if memory pressure exceeds 60% for more than 30 seconds.
|
|
- **Swap Usage**: Monitored at the `/user.slice` level (90% limit).
|
|
- **Verification**: Confirmed via `oomctl` that `/user.slice` is actively monitored with the specified thresholds.
|
|
|
|
### 2. Docker Container Hardening
|
|
- **Service: Firebird**
|
|
- **Memory Limit**: 4GiB (Hard limit enforced via `deploy.resources.limits.memory`).
|
|
- **Current Status**: Running and verified with `docker stats`.
|
|
- **Service: Mailpit**
|
|
- **Memory Limit**: 512MiB (Hard limit enforced via `deploy.resources.limits.memory`).
|
|
- **Current Status**: Running and verified with `docker stats`.
|
|
|
|
### 3. Swap Efficiency (`zswap`)
|
|
- **Status**: Configured in GRUB (`zswap.enabled=1`).
|
|
- **Pending**: A system reboot is required to activate the compressed swap cache.
|
|
- **Current Status**: `N` (Disabled until next boot).
|
|
|
|
## Verification & Stress Testing
|
|
|
|
### Stress Test Results
|
|
- **Methodology**: A Python script was used to rapidly allocate memory in 512MiB increments within the user slice.
|
|
- **Observations**:
|
|
- The system tracked memory growth accurately in `oomctl`.
|
|
- Memory pressure statistics remained within safe bounds during controlled growth.
|
|
- No system-wide hangs occurred during rapid allocation of ~13.5GiB.
|
|
- **Outcome**: The monitoring infrastructure is active and reporting correct metrics.
|
|
|
|
## Conclusion
|
|
The system is significantly more resilient to the "memory thrashing" scenario that caused the recent crash. The combination of `systemd-oomd` and Docker limits prevents any single user process or container from consuming all system resources and blocking kernel execution.
|
|
|
|
**Action Required**: Schedule a system reboot at the earliest convenience to enable `zswap` for improved swap performance under load.
|