ja/mw-macbook-pro
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: Update OpenClaw architecture and VLM prompting guidelines

Browse Source
This commit is contained in:
mw
2026-03-31 17:06:37 +02:00
parent dc18c1cd8b
commit aeeb5faea2
4 changed files with 102 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/OPENCLAW_ARCHITECTURE.md
+26
View File
@@ -47,3 +47,29 @@ The MacBook's node configures the gateway URL as `ws://127.0.0.1:18789` because
### 5. Maintenance Delegation
Note: Specific maintenance and configuration tasks associated with this Mac node—including credentials rotation for tools running here, app updates (`OpenClaw.app`), tunnel configuration, and Mac-based logs (`newsyslog`)—are explicitly delegated to this project repository (`mw-macbook-pro`) and the remote operator logged in via `ssh MWsatwareAG@100.64.0.12`. These responsibilities have been intentionally removed from the Manjaro server's maintenance purview to ensure proper separation of concerns.
### 6. Companion App Configuration (UI Settings)
The macOS OpenClaw Companion App must be configured with specific settings in the "General" tab:
* **Target Mode**: `Remote (another host)`
* **Transport Protocol**: `SSH tunnel`
* **SSH Target**: `root@100.64.0.39:2222` (Tailscale IP of Luca, port 2222 bypassing host into Docker)
* **Gateway Token**: Must be populated with the shared token.
**Feature Approvals**:
The following UI toggles must be **Enabled** for full capability:
* **Allow Canvas**: For displaying agent outputs.
* **Allow Camera**: For agent visual inputs.
* **Enable Peekaboo Bridge**: Critical for native UI automation and vision extraction.
* **Enable debug tools**: Essential for local troubleshooting.
### 7. VLM Prompting Strategy (Qwen 3.5 9B / Qwen-VL)
When utilizing Vision Language Models (VLMs) like Qwen 3.5 9B via the gateway to analyze Mac UIs (e.g., screenshots gathered from Peekaboo), the following strategies are required to mitigate hallucinations:
1. **Structured, Specificity-First Prompting**: Request explicit, format-constrained outputs (e.g., "List every interactive element with its Type, Visible Text, and State") rather than generic descriptions.
2. **Visual Prompts (Image Overlays)**: Ground the VLM's attention using bounding boxes or numbered circles over key UI regions before inference.
3. **Contextual Grounding & Negative Constraints**: Explicitly instruct the model to avoid guessing or inferring hidden functionality. Model must only describe physically visible elements.
4. **Chain-of-Thought (CoT) Visual Reasoning**: Require the model to spatially scan the image (e.g., top to bottom) before answering.
5. **Few-Shot Examples**: Provide example image-answer pairs demonstrating the exact desired structure to establish patterns and dramatically reduce hallucinated elements.