docs: Update OpenClaw architecture and VLM prompting guidelines

This commit is contained in:
mw
2026-03-31 17:06:37 +02:00
parent dc18c1cd8b
commit aeeb5faea2
4 changed files with 102 additions and 0 deletions
+16
View File
@@ -0,0 +1,16 @@
---
description: Episodic memory for build errors (batch-captured)
tags:
- episodic-memory
- build
last_updated: '2026-03-31'
---
# Build Error Patterns (Batch Captured)
## Batch Entry (2026-03-31T14:40:54.767325+00:00)
- **Tool**: execute_command
- **Pattern**: python_module_not_found
- **Output**: `Command failed with exit code 1. Output: Traceback (most recent call last): File "/tmp/ask_remote`
+52
View File
@@ -0,0 +1,52 @@
# OpenClaw macOS Node Configuration & VLM Prompting Research
## Part 1: VLM Prompting Strategy to Mitigate Hallucinations (Qwen 3.5 9B / Qwen-VL)
Based on recent deep research into Vision Language Models (VLMs) and how to prompt them for dense visual interfaces like UI screenshots, the following techniques are required to extract accurate, hallucination-free results from models like Qwen 3.5 9B:
1. **Structured, Specificity-First Prompting**
- **Do not** use generic prompts like "Describe this UI".
- **Do** ask for explicit, format-constrained outputs: *"List every interactive element in this UI. For each element, provide: [Type (e.g., button, input, toggle)], [Visible Text], [State (e.g., active, grayed out)]."*
2. **Visual Prompts (Image Overlays)**
- VLMs perform significantly better when their attention is grounded by visual markers. If possible, preprocess screenshots to add bounding boxes or numbered circles over key UI regions, then prompt the model to describe the contents of "Region 1", "Box 2", etc.
3. **Contextual Grounding & Negative Constraints**
- Explicitly instruct the model to avoid guessing: *"You are a precise UI analyzer. Only describe elements that are physically visible in the screenshot. Do not infer hidden functionality, suggest missing buttons, or guess what dropdowns contain."*
4. **Chain-of-Thought (CoT) Visual Reasoning**
- Ask the model to spatially scan the image before answering: *"First, list the distinct sections of the UI from top to bottom. Then, detail the contents of each section."*
5. **Few-Shot Examples**
- Since Qwen-VL supports few-shot learning, providing one example of an image-answer pair demonstrating the exact desired JSON or markdown structure establishes the expected pattern and dramatically reduces hallucinations.
---
## Part 2: OpenClaw Companion App Configuration Analysis
Based on the provided vision-extracted screenshot (`/tmp/screenshot_desc.md`) and the local architecture documentation (`docs/OPENCLAW_ARCHITECTURE.md`), the OpenClaw macOS companion app is configured as a **Remote Tool Node** connecting to a central Hub.
Here is the exact required configuration for this MacBook (`100.64.0.12`):
### 1. Connection & Transport (The "General" Tab)
* **OpenClaw Status**: Active (Checked)
* **Target Mode**: `Remote (another host)`
* **Transport Protocol**: `SSH tunnel` (Must be selected instead of Direct ws/wss).
* *Reasoning:* OpenClaw's security policy prohibits sending operator tokens over unencrypted websockets to remote IPs. The SSH tunnel wraps the traffic and maps it to a secure local loopback (`127.0.0.1:18789`).
### 2. Network Targeting & Credentials
* **SSH Target**: `root@100.64.0.39:2222`
* *Reasoning:* `100.64.0.39` is the Tailscale mesh IP of the "Luca" server. Port `2222` bypasses the host and connects directly to the SSH daemon running *inside* the `agent-zero` Docker container.
* **Gateway Token**: Must be populated with the shared token. When correctly configured, the UI will display a green `✓ Connected with gateway token` status. The control channel will show `Auth: shared token (gateway/remote.token)`.
### 3. Feature Approvals (Permissions)
As a tool node, the MacBook requires deep macOS integrations. The following feature toggles must be **Enabled (Checked)** in the UI:
* **Allow Canvas**: For displaying agent outputs.
* **Allow Camera**: For agent visual inputs.
* **Enable Peekaboo Bridge**: *Critical* for allowing signed tools (like 'peekaboo') to drive native UI automation on the Mac.
* **Enable debug tools**: Essential for local troubleshooting.
### 4. System-Level Prerequisites
For the configuration above to function, the Mac operating system must grant specific permissions to the underlying Node.js binary (as noted in `OPENCLAW_ARCHITECTURE.md`):
* Accessibility
* Screen Recording
* Full Disk Access
### Summary
The screenshot analyzed by the remote Qwen model perfectly matches the intended architecture. The companion app successfully establishes an encrypted SSH tunnel over Tailscale into the Docker-isolated Hub on Luca, validating its token and registering the Mac as an active node with full Peekaboo UI automation capabilities.
+8
View File
@@ -57,6 +57,14 @@ This repository documents the OpenClaw gateway integration for the satware AG Ma
} }
``` ```
### Companion App UI Configuration
When running via the macOS Companion App, the following settings in the "General" tab are required:
- **Target Mode**: `Remote (another host)`
- **Transport Protocol**: `SSH tunnel`
- **SSH Target**: `root@100.64.0.39:2222`
- **Gateway Token**: Requires the shared gateway token
- **Feature Approvals**: Must enable **Allow Canvas**, **Allow Camera**, **Enable Peekaboo Bridge** (critical for native UI automation), and **Enable debug tools**.
### Security Policy (`~/.openclaw/exec-approvals.json`) ### Security Policy (`~/.openclaw/exec-approvals.json`)
```json ```json
{ {
+26
View File
@@ -47,3 +47,29 @@ The MacBook's node configures the gateway URL as `ws://127.0.0.1:18789` because
### 5. Maintenance Delegation ### 5. Maintenance Delegation
Note: Specific maintenance and configuration tasks associated with this Mac node—including credentials rotation for tools running here, app updates (`OpenClaw.app`), tunnel configuration, and Mac-based logs (`newsyslog`)—are explicitly delegated to this project repository (`mw-macbook-pro`) and the remote operator logged in via `ssh MWsatwareAG@100.64.0.12`. These responsibilities have been intentionally removed from the Manjaro server's maintenance purview to ensure proper separation of concerns. Note: Specific maintenance and configuration tasks associated with this Mac node—including credentials rotation for tools running here, app updates (`OpenClaw.app`), tunnel configuration, and Mac-based logs (`newsyslog`)—are explicitly delegated to this project repository (`mw-macbook-pro`) and the remote operator logged in via `ssh MWsatwareAG@100.64.0.12`. These responsibilities have been intentionally removed from the Manjaro server's maintenance purview to ensure proper separation of concerns.
### 6. Companion App Configuration (UI Settings)
The macOS OpenClaw Companion App must be configured with specific settings in the "General" tab:
* **Target Mode**: `Remote (another host)`
* **Transport Protocol**: `SSH tunnel`
* **SSH Target**: `root@100.64.0.39:2222` (Tailscale IP of Luca, port 2222 bypassing host into Docker)
* **Gateway Token**: Must be populated with the shared token.
**Feature Approvals**:
The following UI toggles must be **Enabled** for full capability:
* **Allow Canvas**: For displaying agent outputs.
* **Allow Camera**: For agent visual inputs.
* **Enable Peekaboo Bridge**: Critical for native UI automation and vision extraction.
* **Enable debug tools**: Essential for local troubleshooting.
### 7. VLM Prompting Strategy (Qwen 3.5 9B / Qwen-VL)
When utilizing Vision Language Models (VLMs) like Qwen 3.5 9B via the gateway to analyze Mac UIs (e.g., screenshots gathered from Peekaboo), the following strategies are required to mitigate hallucinations:
1. **Structured, Specificity-First Prompting**: Request explicit, format-constrained outputs (e.g., "List every interactive element with its Type, Visible Text, and State") rather than generic descriptions.
2. **Visual Prompts (Image Overlays)**: Ground the VLM's attention using bounding boxes or numbered circles over key UI regions before inference.
3. **Contextual Grounding & Negative Constraints**: Explicitly instruct the model to avoid guessing or inferring hidden functionality. Model must only describe physically visible elements.
4. **Chain-of-Thought (CoT) Visual Reasoning**: Require the model to spatially scan the image (e.g., top to bottom) before answering.
5. **Few-Shot Examples**: Provide example image-answer pairs demonstrating the exact desired structure to establish patterns and dramatically reduce hallucinated elements.