From ce99f58374418486baaa2f431dd2b137ef8ddece Mon Sep 17 00:00:00 2001 From: Michael Wegener Date: Mon, 6 Jul 2026 08:58:27 +0200 Subject: [PATCH] docs: Caddy manual override + OnlyOffice MCP SSE investigation (#92) Record the Caddy reverse-proxy at /etc/caddy/ (14 *.localhost sites, mkcert TLS, systemd caddy.service) as a manual override in docs/tech/manual-overrides.md - no caddy Ansible role exists yet. Add a 'Local Dev Services' section to docs/tech/stack.md listing all *.localhost sites and useful Caddy commands. New learning record docs/learnings/2026-07-06-caddy-onlyoffice-mcp-sse- investigation.md: Caddy emits 'context canceled' SSE aborts (~1.5ms) on onlyoffice.localhost/mcp -> localhost:3847. Upstream verified alive (HTTP 406 to plain GET = expected). Handed off to GitLab issue satware/mcp/onlyoffice#92 with 5 hypotheses, 7 tasks, and a proposed flush_interval -1 + transport http { versions 1.1 } mitigation (not applied yet to avoid masking root cause). --- CHANGELOG.md | 4 + ...-caddy-onlyoffice-mcp-sse-investigation.md | 208 ++++++++++++++++++ docs/tech/manual-overrides.md | 3 +- docs/tech/stack.md | 43 +++- 4 files changed, 256 insertions(+), 2 deletions(-) create mode 100644 docs/learnings/2026-07-06-caddy-onlyoffice-mcp-sse-investigation.md diff --git a/CHANGELOG.md b/CHANGELOG.md index e002354..bdf2d13 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Added +- Documented local dev services in `docs/tech/stack.md`: Caddy (systemd `caddy.service` on port 443) with 14 `*.localhost` sites and mkcert TLS at `/etc/caddy/`. Registered as a manual override in `docs/tech/manual-overrides.md` (no `caddy` Ansible role exists yet) (2026-07-06). +- Investigation learning record `docs/learnings/2026-07-06-caddy-onlyoffice-mcp-sse-investigation.md`: Caddy `context canceled` SSE aborts (~1.5ms) on `onlyoffice.localhost/mcp` forwarded to `localhost:3847` (OnlyOffice MCP server). Upstream verified alive (HTTP 406 to plain GET = expected). Handed off to GitLab issue [satware/mcp/onlyoffice#92](https://gitlab.satware.com/satware/mcp/onlyoffice/-/work_items/92) with 5 hypotheses, 7 investigation tasks, and a proposed Caddy `flush_interval -1` + `transport http { versions 1.1 }` mitigation (not applied yet) (2026-07-06). + ### Fixed - tuigreet boot error: removed invalid `--numlock` flag from greetd config (not a valid tuigreet parameter). Replaced X11-only `numlockx on` in Hyprland with native `numlock_by_default = true` in the `input` section (2026-06-01). - hypridle SEGV crash loop (ADR N/A, hotfix): sdbus-cpp 2.3.0 had an ABI incompatibility causing hypridle to SIGSEGV in `libsdbus-c++.so.2` `registerSignalHandler`. Fixed by upgrading sdbus-cpp from 2.3.0-1 to 2.3.1-1 (Arch upstream, pulled ahead of Manjaro mirror sync). Also resolves latent ABI issues for hyprlock and xdg-desktop-portal-hyprland (2026-05-22). diff --git a/docs/learnings/2026-07-06-caddy-onlyoffice-mcp-sse-investigation.md b/docs/learnings/2026-07-06-caddy-onlyoffice-mcp-sse-investigation.md new file mode 100644 index 0000000..bf2699a --- /dev/null +++ b/docs/learnings/2026-07-06-caddy-onlyoffice-mcp-sse-investigation.md @@ -0,0 +1,208 @@ +--- +description: >- + Session record (2026-07-06): triaged Caddy journald warnings on + onlyoffice.localhost/mcp showing "context canceled" SSE aborts (~1.5ms) + from opencode 1.17.13 MCP client against the OnlyOffice MCP server on + :3847. Verified upstream liveness, opened GitLab issue #92 in + satware/mcp/onlyoffice, and documented the Caddy reverse-proxy as a + manual override. +tags: + - learning + - caddy + - onlyoffice + - mcp + - sse + - reverse-proxy + - investigation +last_updated: '2026-07-06' +--- + +# Caddy "context canceled" on onlyoffice.localhost/mcp - investigation - 2026-07-06 + +## Summary + +Caddy (systemd `caddy.service`) on `mw-manjaro-pf` is emitting a high +volume of `warn`-level journald entries: `aborting with incomplete +response ... error: "reading: context canceled"` for `GET /mcp` requests +on `https://onlyoffice.localhost`. Each abort fires ~1.5ms after the +request is forwarded to the upstream `localhost:3847` (OnlyOffice MCP +server). The upstream is alive and correctly rejects non-SSE GETs with +HTTP 406, so this is a stream-lifecycle issue, not a server-down issue. + +Investigation handed off to the OnlyOffice MCP repo as GitLab issue #92. +This workstation's Caddy reverse-proxy config is now recorded as a manual +override (not Ansible-managed). + +## Incident + +| Field | Value | +|-------|-------| +| Symptom | Repeated `context canceled` warnings in `journalctl -u caddy` | +| Caddy site | `https://onlyoffice.localhost` -> `reverse_proxy localhost:3847` | +| Upstream | `localhost:3847` (OnlyOffice MCP server) | +| Request | `GET /mcp` with `Accept: text/event-stream` | +| Client | `opencode/1.17.13` (MCP streamable-HTTP transport) | +| MCP protocol | `2025-11-25` | +| Abort duration | ~1.5ms (request forwarded, then read context canceled) | +| Frequency | Multiple per minute, across distinct `Mcp-Session-Id` values | + +## Evidence + +### Caddy log sample (recurring) + +```json +{ + "level": "warn", + "logger": "http.handlers.reverse_proxy", + "msg": "aborting with incomplete response", + "upstream": "localhost:3847", + "duration": 0.00173631, + "request": { + "method": "GET", + "host": "onlyoffice.localhost", + "uri": "/mcp", + "headers": { + "Accept": ["text/event-stream"], + "Mcp-Protocol-Version": ["2025-11-25"], + "Mcp-Session-Id": ["a5e859b4-01f1-4c24-81bb-78ee02d74357"], + "User-Agent": ["opencode/1.17.13"], + "Via": ["1.1 Caddy"] + } + }, + "error": "reading: context canceled" +} +``` + +Observed `Mcp-Session-Id` values (each retried over several minutes): +`a5e859b4...`, `4bfbb912...`, `3c4a0454...`, `08397c3d...`, +`ed762f08...`, `b0fae341...`. + +### Upstream liveness verification + +```bash +$ ss -tlnp | grep :3847 +LISTEN 0 4096 0.0.0.0:3847 0.0.0.0:* + +$ curl -sS -o /dev/null -w "HTTP %{http_code} in %{time_total}s\n" \ + --max-time 3 http://localhost:3847/mcp +HTTP 406 in 0.004665s +``` + +HTTP 406 is expected for a plain GET without the required SSE headers +the MCP server is alive and rejecting non-SSE requests correctly. + +## Diagnosis + +### Root cause (not yet confirmed - under investigation) + +Five candidate hypotheses, listed in the GitLab issue. The leading +suspect is **client-side disconnect**: opencode 1.17.13 opens the SSE +stream then immediately closes it (the ~1.5ms duration is too short for +a genuine read). Secondary suspects are server-side stream teardown, +MCP protocol-version mismatch (`2025-11-25`), Caddy HTTP/1.1 framing of +SSE, or a client session-pool bug creating and abandoning sessions. + +### Why this workstation's concern + +The Caddy reverse-proxy is the boundary that surfaces the symptom. Even +if the root cause is client- or server-side, the local config matters +because: + +1. Caddy logs are being polluted with warn-level noise (masks real errors). +2. The reverse-proxy block has no SSE-specific tuning (`flush_interval`, + explicit HTTP/1.1 transport) that could mask or mitigate upstream + misbehavior. +3. The Caddy config at `/etc/caddy/conf.d/localhost-dev.conf` is **not** + Ansible-managed by this repo - it is a manual override that must be + recorded (see `docs/tech/manual-overrides.md`). + +## Environment + +| Component | Version / Value | +|-----------|----------------| +| Host | `mw-manjaro-pf` (mw-pfeddersheim-workstation) | +| OS | Manjaro Linux, Wayland session | +| Caddy | systemd `caddy.service`, active since 2026-07-02 | +| Caddy config | `/etc/caddy/Caddyfile` + `/etc/caddy/conf.d/localhost-dev.conf` | +| TLS cert | mkcert pair at `/etc/caddy/localhost{,-key}.pem` | +| Upstream | `localhost:3847` (OnlyOffice MCP server, process TBD) | +| MCP client | `opencode/1.17.13` | +| MCP protocol | `2025-11-25` (streamable HTTP transport) | + +## *.localhost sites configured + +All defined in `/etc/caddy/conf.d/localhost-dev.conf` (14 sites total): + +| Site | Backend / Handler | +|------|-------------------| +| `mcp.localhost` | `reverse_proxy localhost:50880` | +| `artifacts.localhost` | `file_server browse` root `/var/www/artifacts` | +| `onlyoffice.localhost` | `reverse_proxy localhost:3847` (subject of this issue) | +| `project-1.localhost` ... `project-10.localhost` | `respond "project-N - placeholder"` (10 reserved) | +| `demo.satware.com.localhost` | `reverse_proxy localhost:38081` (webdevops/php-apache-dev container) | + +## Handoff + +### GitLab issue + +| Item | Value | +|------|-------| +| Project | `satware/mcp/onlyoffice` (`gitlab.satware.com`) | +| Issue | [#92](https://gitlab.satware.com/satware/mcp/onlyoffice/-/work_items/92) | +| Title | Investigation: Caddy reports 'context canceled' on onlyoffice.localhost/mcp SSE stream (~1.5ms aborts) | +| Labels | `bug`, `investigation`, `mcp` | +| State | open | + +The issue body contains the full evidence, 5 hypotheses, 7 investigation +tasks, and a proposed Caddy mitigation (`flush_interval -1` + +`transport http { versions 1.1 }`). + +### Next actions (assigned to OnlyOffice MCP repo) + +1. Identify the process listening on `:3847` (`sudo ss -tlnp | grep 3847`). +2. Collect server-side logs for the `Mcp-Session-Id` values cited above. +3. Reproduce with `curl -N -H "Accept: text/event-stream" -H "Mcp-Protocol-Version: 2025-11-25" https://onlyoffice.localhost/mcp`. +4. Check opencode 1.17.13 changelog for MCP SSE session bugs. +5. Verify Caddy `reverse_proxy` SSE tuning (proposed block in issue #92). +6. Confirm whether `notifications/initialized` round-trip completes. +7. Log the negotiated MCP protocol version server-side. + +## Local documentation updates + +| File | Change | +|------|-------| +| `docs/tech/manual-overrides.md` | Added row for `/etc/caddy/` config (not Ansible-managed) | +| `docs/tech/stack.md` | Added "Local Dev Services" section listing Caddy + `*.localhost` sites | +| `CHANGELOG.md` | Added `[Unreleased] -> Added` entry | +| GitLab issue #92 | Created with full evidence + hypotheses + tasks | + +## Proposed Caddy mitigation (to test on this workstation) + +```caddyfile +https://onlyoffice.localhost { + tls /etc/caddy/localhost.pem /etc/caddy/localhost-key.pem + reverse_proxy localhost:3847 { + flush_interval -1 + transport http { + versions 1.1 + } + } +} +``` + +`flush_interval -1` disables Caddy's buffering so SSE events flush to the +client immediately; `transport http { versions 1.1 }` pins the upstream +leg to HTTP/1.1 (Caddy logs already show HTTP/1.1 for the upstream, this +makes it explicit and avoids HTTP/2 SSE framing surprises). + +Not applied yet - pending GitLab issue #92 resolution to avoid masking +the root cause. + +## References + +- GitLab issue: https://gitlab.satware.com/satware/mcp/onlyoffice/-/work_items/92 +- Caddy config: `/etc/caddy/conf.d/localhost-dev.conf` +- Local skill: `caddy` (`~/.config/opencode/skills/caddy/SKILL.md`) +- MCP transport spec: https://modelcontextprotocol.io/specification/2025-11-25/basic/transports +- Manual overrides: `docs/tech/manual-overrides.md` +- Software stack: `docs/tech/stack.md` diff --git a/docs/tech/manual-overrides.md b/docs/tech/manual-overrides.md index e4d84b0..a1762f9 100644 --- a/docs/tech/manual-overrides.md +++ b/docs/tech/manual-overrides.md @@ -6,7 +6,7 @@ tags: - manual-overrides - infrastructure - configuration -last_updated: '2026-06-17' +last_updated: '2026-07-06' --- # Manual System Overrides @@ -22,6 +22,7 @@ This document tracks all persistent system changes implemented manually that are | Date | Change | Rationale | Ansible Status | |------|--------|-----------|----------------| +| 2026-07-06 | Caddy reverse-proxy + mkcert TLS at `/etc/caddy/` (`Caddyfile` + `conf.d/localhost-dev.conf` + `localhost{,-key}.pem`) | Local HTTPS dev for 14 `*.localhost` sites (mcp, artifacts, onlyoffice, demo.satware.com clone, 10 placeholders). Caddy runs as systemd `caddy.service` on port 443. NOT managed by this repo - no `caddy` Ansible role exists. The `onlyoffice.localhost` site (-> `localhost:3847`) is emitting `context canceled` SSE aborts; see `docs/learnings/2026-07-06-caddy-onlyoffice-mcp-sse-investigation.md` and GitLab issue [satware/mcp/onlyoffice#92](https://gitlab.satware.com/satware/mcp/onlyoffice/-/work_items/92). | Candidate for Ansible role (`caddy` + `mkcert`); defer until issue #92 resolves to avoid masking root cause | | 2026-07-03 | Patched `remmina-plugin-rdp.so` in `~/.config/remmina/plugins/` | Remmina 1.4.43 crashes (SIGSEGV) on Wayland when `keymap` is set in a profile. Built from `~/external/remmina/` fork with NULL check fix. Alt plugin dir loads before system plugin. MR !2757 submitted upstream. See `docs/learnings/2026-07-03-remmina-wayland-crash-fix.md`. | Remove when upstream fix is released and packaged | | 2026-06-17 | Discovered and deleted `br_vm_internal` bridge (10.10.2.0/24) | Externally-created bridge, no NM config file, no VMs attached. Origin unknown — likely ad-hoc `nmcli con add` or manual `brctl` session (Jun 10). Traffic-shaping qdisc (`htb`) suggests prior VM network isolation attempt. Deleted via `nmcli con delete` + `ip link delete`. | N/A (removed) | | 2026-04-02 | Moved LLM models to `/home/mw/models/` (SATA SSD) with symlinks | Free root NVMe space (93%→87%, +25G freed) | Not yet | diff --git a/docs/tech/stack.md b/docs/tech/stack.md index b4fda8e..879efae 100644 --- a/docs/tech/stack.md +++ b/docs/tech/stack.md @@ -6,7 +6,7 @@ tags: - software-stack - inventory - tech-stack -last_updated: '2026-06-17' +last_updated: '2026-07-06' --- # Software Stack - mw-pfeddersheim-workstation @@ -92,3 +92,44 @@ Packages previously listed as "Misc" reclassified to functional domains: - **Ansible**: `/usr/bin/ansible` - **Python**: `/usr/bin/python` - **Docker**: `/usr/bin/docker` + +## Local Dev Services (Manual - not Ansible-managed) + +System-level services running on this workstation that are configured manually +outside this repo's Ansible roles. See `docs/tech/manual-overrides.md` for the +full override table and `docs/learnings/2026-07-06-caddy-onlyoffice-mcp-sse-investigation.md` +for the active Caddy/OnlyOffice investigation. + +### Caddy (port 443, systemd `caddy.service`) + +| Item | Value | +|------|-------| +| Package | `caddy` (pacman) | +| Service | `caddy.service` (enabled, runs as `caddy` user) | +| Config | `/etc/caddy/Caddyfile` + `/etc/caddy/conf.d/localhost-dev.conf` | +| TLS | mkcert-generated pair at `/etc/caddy/localhost{,-key}.pem` | +| Admin endpoint | Unix socket `/run/caddy/admin.socket` | +| Sites | 14 `*.localhost` virtual hosts (see table below) | + +#### `*.localhost` sites + +| Site | Backend / Handler | Notes | +|------|-------------------|-------| +| `mcp.localhost` | `reverse_proxy localhost:50880` | MCP server (node, pid 1167) | +| `artifacts.localhost` | `file_server browse` root `/var/www/artifacts` | Static artifacts | +| `onlyoffice.localhost` | `reverse_proxy localhost:3847` | OnlyOffice MCP server; emitting `context canceled` SSE aborts (issue [#92](https://gitlab.satware.com/satware/mcp/onlyoffice/-/work_items/92)) | +| `project-1.localhost` ... `project-10.localhost` | `respond "project-N - placeholder"` | 10 reserved placeholders | +| `demo.satware.com.localhost` | `reverse_proxy localhost:38081` | webdevops/php-apache-dev container | + +Wildcard cert covers one level only. `demo.satware.com.localhost` works +(single label under `*.localhost`); deeper nesting (e.g. +`sub.demo.satware.com.localhost`) would NOT be covered. + +### Useful Caddy commands + +```bash +sudo caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile +sudo systemctl reload caddy +sudo journalctl -u caddy --no-pager -n 50 +curl -sk https://onlyoffice.localhost/ping # test (skips cert verify) +```