infrastructure/mw-pfeddersheim-workstation
2
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: Caddy manual override + OnlyOffice MCP SSE investigation (#92)

Browse Source 
Record the Caddy reverse-proxy at /etc/caddy/ (14 *.localhost sites,
mkcert TLS, systemd caddy.service) as a manual override in
docs/tech/manual-overrides.md - no caddy Ansible role exists yet.

Add a 'Local Dev Services' section to docs/tech/stack.md listing all
*.localhost sites and useful Caddy commands.

New learning record docs/learnings/2026-07-06-caddy-onlyoffice-mcp-sse-
investigation.md: Caddy emits 'context canceled' SSE aborts (~1.5ms) on
onlyoffice.localhost/mcp -> localhost:3847. Upstream verified alive
(HTTP 406 to plain GET = expected). Handed off to GitLab issue
satware/mcp/onlyoffice#92 with 5 hypotheses, 7 tasks, and a proposed
flush_interval -1 + transport http { versions 1.1 } mitigation (not
applied yet to avoid masking root cause).
This commit is contained in:
mw
2026-07-06 08:58:27 +02:00
parent 2b6ce0d635
commit ce99f58374
4 changed files with 256 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+4
View File
@@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased] ## [Unreleased]
### Added
- Documented local dev services in `docs/tech/stack.md`: Caddy (systemd `caddy.service` on port 443) with 14 `*.localhost` sites and mkcert TLS at `/etc/caddy/`. Registered as a manual override in `docs/tech/manual-overrides.md` (no `caddy` Ansible role exists yet) (2026-07-06).
- Investigation learning record `docs/learnings/2026-07-06-caddy-onlyoffice-mcp-sse-investigation.md`: Caddy `context canceled` SSE aborts (~1.5ms) on `onlyoffice.localhost/mcp` forwarded to `localhost:3847` (OnlyOffice MCP server). Upstream verified alive (HTTP 406 to plain GET = expected). Handed off to GitLab issue [satware/mcp/onlyoffice#92](https://gitlab.satware.com/satware/mcp/onlyoffice/-/work_items/92) with 5 hypotheses, 7 investigation tasks, and a proposed Caddy `flush_interval -1` + `transport http { versions 1.1 }` mitigation (not applied yet) (2026-07-06).
### Fixed ### Fixed
- tuigreet boot error: removed invalid `--numlock` flag from greetd config (not a valid tuigreet parameter). Replaced X11-only `numlockx on` in Hyprland with native `numlock_by_default = true` in the `input` section (2026-06-01). - tuigreet boot error: removed invalid `--numlock` flag from greetd config (not a valid tuigreet parameter). Replaced X11-only `numlockx on` in Hyprland with native `numlock_by_default = true` in the `input` section (2026-06-01).
- hypridle SEGV crash loop (ADR N/A, hotfix): sdbus-cpp 2.3.0 had an ABI incompatibility causing hypridle to SIGSEGV in `libsdbus-c++.so.2` `registerSignalHandler`. Fixed by upgrading sdbus-cpp from 2.3.0-1 to 2.3.1-1 (Arch upstream, pulled ahead of Manjaro mirror sync). Also resolves latent ABI issues for hyprlock and xdg-desktop-portal-hyprland (2026-05-22). - hypridle SEGV crash loop (ADR N/A, hotfix): sdbus-cpp 2.3.0 had an ABI incompatibility causing hypridle to SIGSEGV in `libsdbus-c++.so.2` `registerSignalHandler`. Fixed by upgrading sdbus-cpp from 2.3.0-1 to 2.3.1-1 (Arch upstream, pulled ahead of Manjaro mirror sync). Also resolves latent ABI issues for hyprlock and xdg-desktop-portal-hyprland (2026-05-22).
docs/learnings/2026-07-06-caddy-onlyoffice-mcp-sse-investigation.md
+208
View File
@@ -0,0 +1,208 @@
---
description: >-
Session record (2026-07-06): triaged Caddy journald warnings on
onlyoffice.localhost/mcp showing "context canceled" SSE aborts (~1.5ms)
from opencode 1.17.13 MCP client against the OnlyOffice MCP server on
:3847. Verified upstream liveness, opened GitLab issue #92 in
satware/mcp/onlyoffice, and documented the Caddy reverse-proxy as a
manual override.
tags:
- learning
- caddy
- onlyoffice
- mcp
- sse
- reverse-proxy
- investigation
last_updated: '2026-07-06'
---
# Caddy "context canceled" on onlyoffice.localhost/mcp - investigation - 2026-07-06
## Summary
Caddy (systemd `caddy.service`) on `mw-manjaro-pf` is emitting a high
volume of `warn`-level journald entries: `aborting with incomplete
response ... error: "reading: context canceled"` for `GET /mcp` requests
on `https://onlyoffice.localhost`. Each abort fires ~1.5ms after the
request is forwarded to the upstream `localhost:3847` (OnlyOffice MCP
server). The upstream is alive and correctly rejects non-SSE GETs with
HTTP 406, so this is a stream-lifecycle issue, not a server-down issue.
Investigation handed off to the OnlyOffice MCP repo as GitLab issue #92.
This workstation's Caddy reverse-proxy config is now recorded as a manual
override (not Ansible-managed).
## Incident
| Field | Value |
|-------|-------|
| Symptom | Repeated `context canceled` warnings in `journalctl -u caddy` |
| Caddy site | `https://onlyoffice.localhost` -> `reverse_proxy localhost:3847` |
| Upstream | `localhost:3847` (OnlyOffice MCP server) |
| Request | `GET /mcp` with `Accept: text/event-stream` |
| Client | `opencode/1.17.13` (MCP streamable-HTTP transport) |
| MCP protocol | `2025-11-25` |
| Abort duration | ~1.5ms (request forwarded, then read context canceled) |
| Frequency | Multiple per minute, across distinct `Mcp-Session-Id` values |
## Evidence
### Caddy log sample (recurring)
```json
{
"level": "warn",
"logger": "http.handlers.reverse_proxy",
"msg": "aborting with incomplete response",
"upstream": "localhost:3847",
"duration": 0.00173631,
"request": {
"method": "GET",
"host": "onlyoffice.localhost",
"uri": "/mcp",
"headers": {
"Accept": ["text/event-stream"],
"Mcp-Protocol-Version": ["2025-11-25"],
"Mcp-Session-Id": ["a5e859b4-01f1-4c24-81bb-78ee02d74357"],
"User-Agent": ["opencode/1.17.13"],
"Via": ["1.1 Caddy"]
}
},
"error": "reading: context canceled"
}
```
Observed `Mcp-Session-Id` values (each retried over several minutes):
`a5e859b4...`, `4bfbb912...`, `3c4a0454...`, `08397c3d...`,
`ed762f08...`, `b0fae341...`.
### Upstream liveness verification
```bash
$ ss -tlnp | grep :3847
LISTEN 0 4096 0.0.0.0:3847 0.0.0.0:*
$ curl -sS -o /dev/null -w "HTTP %{http_code} in %{time_total}s\n" \
--max-time 3 http://localhost:3847/mcp
HTTP 406 in 0.004665s
```
HTTP 406 is expected for a plain GET without the required SSE headers
the MCP server is alive and rejecting non-SSE requests correctly.
## Diagnosis
### Root cause (not yet confirmed - under investigation)
Five candidate hypotheses, listed in the GitLab issue. The leading
suspect is **client-side disconnect**: opencode 1.17.13 opens the SSE
stream then immediately closes it (the ~1.5ms duration is too short for
a genuine read). Secondary suspects are server-side stream teardown,
MCP protocol-version mismatch (`2025-11-25`), Caddy HTTP/1.1 framing of
SSE, or a client session-pool bug creating and abandoning sessions.
### Why this workstation's concern
The Caddy reverse-proxy is the boundary that surfaces the symptom. Even
if the root cause is client- or server-side, the local config matters
because:
1. Caddy logs are being polluted with warn-level noise (masks real errors).
2. The reverse-proxy block has no SSE-specific tuning (`flush_interval`,
explicit HTTP/1.1 transport) that could mask or mitigate upstream
misbehavior.
3. The Caddy config at `/etc/caddy/conf.d/localhost-dev.conf` is **not**
Ansible-managed by this repo - it is a manual override that must be
recorded (see `docs/tech/manual-overrides.md`).
## Environment
| Component | Version / Value |
|-----------|----------------|
| Host | `mw-manjaro-pf` (mw-pfeddersheim-workstation) |
| OS | Manjaro Linux, Wayland session |
| Caddy | systemd `caddy.service`, active since 2026-07-02 |
| Caddy config | `/etc/caddy/Caddyfile` + `/etc/caddy/conf.d/localhost-dev.conf` |
| TLS cert | mkcert pair at `/etc/caddy/localhost{,-key}.pem` |
| Upstream | `localhost:3847` (OnlyOffice MCP server, process TBD) |
| MCP client | `opencode/1.17.13` |
| MCP protocol | `2025-11-25` (streamable HTTP transport) |
## *.localhost sites configured
All defined in `/etc/caddy/conf.d/localhost-dev.conf` (14 sites total):
| Site | Backend / Handler |
|------|-------------------|
| `mcp.localhost` | `reverse_proxy localhost:50880` |
| `artifacts.localhost` | `file_server browse` root `/var/www/artifacts` |
| `onlyoffice.localhost` | `reverse_proxy localhost:3847` (subject of this issue) |
| `project-1.localhost` ... `project-10.localhost` | `respond "project-N - placeholder"` (10 reserved) |
| `demo.satware.com.localhost` | `reverse_proxy localhost:38081` (webdevops/php-apache-dev container) |
## Handoff
### GitLab issue
| Item | Value |
|------|-------|
| Project | `satware/mcp/onlyoffice` (`gitlab.satware.com`) |
| Issue | [#92](https://gitlab.satware.com/satware/mcp/onlyoffice/-/work_items/92) |
| Title | Investigation: Caddy reports 'context canceled' on onlyoffice.localhost/mcp SSE stream (~1.5ms aborts) |
| Labels | `bug`, `investigation`, `mcp` |
| State | open |
The issue body contains the full evidence, 5 hypotheses, 7 investigation
tasks, and a proposed Caddy mitigation (`flush_interval -1` +
`transport http { versions 1.1 }`).
### Next actions (assigned to OnlyOffice MCP repo)
1. Identify the process listening on `:3847` (`sudo ss -tlnp | grep 3847`).
2. Collect server-side logs for the `Mcp-Session-Id` values cited above.
3. Reproduce with `curl -N -H "Accept: text/event-stream" -H "Mcp-Protocol-Version: 2025-11-25" https://onlyoffice.localhost/mcp`.
4. Check opencode 1.17.13 changelog for MCP SSE session bugs.
5. Verify Caddy `reverse_proxy` SSE tuning (proposed block in issue #92).
6. Confirm whether `notifications/initialized` round-trip completes.
7. Log the negotiated MCP protocol version server-side.
## Local documentation updates
| File | Change |
|------|-------|
| `docs/tech/manual-overrides.md` | Added row for `/etc/caddy/` config (not Ansible-managed) |
| `docs/tech/stack.md` | Added "Local Dev Services" section listing Caddy + `*.localhost` sites |
| `CHANGELOG.md` | Added `[Unreleased] -> Added` entry |
| GitLab issue #92 | Created with full evidence + hypotheses + tasks |
## Proposed Caddy mitigation (to test on this workstation)
```caddyfile
https://onlyoffice.localhost {
tls /etc/caddy/localhost.pem /etc/caddy/localhost-key.pem
reverse_proxy localhost:3847 {
flush_interval -1
transport http {
versions 1.1
}
}
}
```
`flush_interval -1` disables Caddy's buffering so SSE events flush to the
client immediately; `transport http { versions 1.1 }` pins the upstream
leg to HTTP/1.1 (Caddy logs already show HTTP/1.1 for the upstream, this
makes it explicit and avoids HTTP/2 SSE framing surprises).
Not applied yet - pending GitLab issue #92 resolution to avoid masking
the root cause.
## References
- GitLab issue: https://gitlab.satware.com/satware/mcp/onlyoffice/-/work_items/92
- Caddy config: `/etc/caddy/conf.d/localhost-dev.conf`
- Local skill: `caddy` (`~/.config/opencode/skills/caddy/SKILL.md`)
- MCP transport spec: https://modelcontextprotocol.io/specification/2025-11-25/basic/transports
- Manual overrides: `docs/tech/manual-overrides.md`
- Software stack: `docs/tech/stack.md`
docs/tech/manual-overrides.md
+2 -1
View File
@@ -6,7 +6,7 @@ tags:
- manual-overrides - manual-overrides
- infrastructure - infrastructure
- configuration - configuration
last_updated: '2026-06-17' last_updated: '2026-07-06'
--- ---
# Manual System Overrides # Manual System Overrides
@@ -22,6 +22,7 @@ This document tracks all persistent system changes implemented manually that are
| Date | Change | Rationale | Ansible Status | | Date | Change | Rationale | Ansible Status |
|------|--------|-----------|----------------| |------|--------|-----------|----------------|
| 2026-07-06 | Caddy reverse-proxy + mkcert TLS at `/etc/caddy/` (`Caddyfile` + `conf.d/localhost-dev.conf` + `localhost{,-key}.pem`) | Local HTTPS dev for 14 `*.localhost` sites (mcp, artifacts, onlyoffice, demo.satware.com clone, 10 placeholders). Caddy runs as systemd `caddy.service` on port 443. NOT managed by this repo - no `caddy` Ansible role exists. The `onlyoffice.localhost` site (-> `localhost:3847`) is emitting `context canceled` SSE aborts; see `docs/learnings/2026-07-06-caddy-onlyoffice-mcp-sse-investigation.md` and GitLab issue [satware/mcp/onlyoffice#92](https://gitlab.satware.com/satware/mcp/onlyoffice/-/work_items/92). | Candidate for Ansible role (`caddy` + `mkcert`); defer until issue #92 resolves to avoid masking root cause |
| 2026-07-03 | Patched `remmina-plugin-rdp.so` in `~/.config/remmina/plugins/` | Remmina 1.4.43 crashes (SIGSEGV) on Wayland when `keymap` is set in a profile. Built from `~/external/remmina/` fork with NULL check fix. Alt plugin dir loads before system plugin. MR !2757 submitted upstream. See `docs/learnings/2026-07-03-remmina-wayland-crash-fix.md`. | Remove when upstream fix is released and packaged | | 2026-07-03 | Patched `remmina-plugin-rdp.so` in `~/.config/remmina/plugins/` | Remmina 1.4.43 crashes (SIGSEGV) on Wayland when `keymap` is set in a profile. Built from `~/external/remmina/` fork with NULL check fix. Alt plugin dir loads before system plugin. MR !2757 submitted upstream. See `docs/learnings/2026-07-03-remmina-wayland-crash-fix.md`. | Remove when upstream fix is released and packaged |
| 2026-06-17 | Discovered and deleted `br_vm_internal` bridge (10.10.2.0/24) | Externally-created bridge, no NM config file, no VMs attached. Origin unknown — likely ad-hoc `nmcli con add` or manual `brctl` session (Jun 10). Traffic-shaping qdisc (`htb`) suggests prior VM network isolation attempt. Deleted via `nmcli con delete` + `ip link delete`. | N/A (removed) | | 2026-06-17 | Discovered and deleted `br_vm_internal` bridge (10.10.2.0/24) | Externally-created bridge, no NM config file, no VMs attached. Origin unknown — likely ad-hoc `nmcli con add` or manual `brctl` session (Jun 10). Traffic-shaping qdisc (`htb`) suggests prior VM network isolation attempt. Deleted via `nmcli con delete` + `ip link delete`. | N/A (removed) |
| 2026-04-02 | Moved LLM models to `/home/mw/models/` (SATA SSD) with symlinks | Free root NVMe space (93%→87%, +25G freed) | Not yet | | 2026-04-02 | Moved LLM models to `/home/mw/models/` (SATA SSD) with symlinks | Free root NVMe space (93%→87%, +25G freed) | Not yet |
docs/tech/stack.md
+42 -1
View File
@@ -6,7 +6,7 @@ tags:
- software-stack - software-stack
- inventory - inventory
- tech-stack - tech-stack
last_updated: '2026-06-17' last_updated: '2026-07-06'
--- ---
# Software Stack - mw-pfeddersheim-workstation # Software Stack - mw-pfeddersheim-workstation
@@ -92,3 +92,44 @@ Packages previously listed as "Misc" reclassified to functional domains:
- **Ansible**: `/usr/bin/ansible` - **Ansible**: `/usr/bin/ansible`
- **Python**: `/usr/bin/python` - **Python**: `/usr/bin/python`
- **Docker**: `/usr/bin/docker` - **Docker**: `/usr/bin/docker`
## Local Dev Services (Manual - not Ansible-managed)
System-level services running on this workstation that are configured manually
outside this repo's Ansible roles. See `docs/tech/manual-overrides.md` for the
full override table and `docs/learnings/2026-07-06-caddy-onlyoffice-mcp-sse-investigation.md`
for the active Caddy/OnlyOffice investigation.
### Caddy (port 443, systemd `caddy.service`)
| Item | Value |
|------|-------|
| Package | `caddy` (pacman) |
| Service | `caddy.service` (enabled, runs as `caddy` user) |
| Config | `/etc/caddy/Caddyfile` + `/etc/caddy/conf.d/localhost-dev.conf` |
| TLS | mkcert-generated pair at `/etc/caddy/localhost{,-key}.pem` |
| Admin endpoint | Unix socket `/run/caddy/admin.socket` |
| Sites | 14 `*.localhost` virtual hosts (see table below) |
#### `*.localhost` sites
| Site | Backend / Handler | Notes |
|------|-------------------|-------|
| `mcp.localhost` | `reverse_proxy localhost:50880` | MCP server (node, pid 1167) |
| `artifacts.localhost` | `file_server browse` root `/var/www/artifacts` | Static artifacts |
| `onlyoffice.localhost` | `reverse_proxy localhost:3847` | OnlyOffice MCP server; emitting `context canceled` SSE aborts (issue [#92](https://gitlab.satware.com/satware/mcp/onlyoffice/-/work_items/92)) |
| `project-1.localhost` ... `project-10.localhost` | `respond "project-N - placeholder"` | 10 reserved placeholders |
| `demo.satware.com.localhost` | `reverse_proxy localhost:38081` | webdevops/php-apache-dev container |
Wildcard cert covers one level only. `demo.satware.com.localhost` works
(single label under `*.localhost`); deeper nesting (e.g.
`sub.demo.satware.com.localhost`) would NOT be covered.
### Useful Caddy commands
```bash
sudo caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile
sudo systemctl reload caddy
sudo journalctl -u caddy --no-pager -n 50
curl -sk https://onlyoffice.localhost/ping # test (skips cert verify)
```