From aeeb5faea2d2ef4a5afdf0a64b05ff73839082f9 Mon Sep 17 00:00:00 2001 From: Michael Wegener Date: Tue, 31 Mar 2026 17:06:37 +0200 Subject: [PATCH] docs: Update OpenClaw architecture and VLM prompting guidelines --- .clinerules/build-patterns.md | 16 ++++++++++ OPENCLAW_CONFIGURATION_REPORT.md | 52 ++++++++++++++++++++++++++++++++ README.md | 8 +++++ docs/OPENCLAW_ARCHITECTURE.md | 26 ++++++++++++++++ 4 files changed, 102 insertions(+) create mode 100644 .clinerules/build-patterns.md create mode 100644 OPENCLAW_CONFIGURATION_REPORT.md diff --git a/.clinerules/build-patterns.md b/.clinerules/build-patterns.md new file mode 100644 index 0000000..f4b7265 --- /dev/null +++ b/.clinerules/build-patterns.md @@ -0,0 +1,16 @@ +--- +description: Episodic memory for build errors (batch-captured) +tags: + - episodic-memory + - build +last_updated: '2026-03-31' +--- + +# Build Error Patterns (Batch Captured) + +## Batch Entry (2026-03-31T14:40:54.767325+00:00) + +- **Tool**: execute_command +- **Pattern**: python_module_not_found +- **Output**: `Command failed with exit code 1. Output: Traceback (most recent call last): File "/tmp/ask_remote` + diff --git a/OPENCLAW_CONFIGURATION_REPORT.md b/OPENCLAW_CONFIGURATION_REPORT.md new file mode 100644 index 0000000..5273629 --- /dev/null +++ b/OPENCLAW_CONFIGURATION_REPORT.md @@ -0,0 +1,52 @@ +# OpenClaw macOS Node Configuration & VLM Prompting Research + +## Part 1: VLM Prompting Strategy to Mitigate Hallucinations (Qwen 3.5 9B / Qwen-VL) + +Based on recent deep research into Vision Language Models (VLMs) and how to prompt them for dense visual interfaces like UI screenshots, the following techniques are required to extract accurate, hallucination-free results from models like Qwen 3.5 9B: + +1. **Structured, Specificity-First Prompting** + - **Do not** use generic prompts like "Describe this UI". + - **Do** ask for explicit, format-constrained outputs: *"List every interactive element in this UI. For each element, provide: [Type (e.g., button, input, toggle)], [Visible Text], [State (e.g., active, grayed out)]."* +2. **Visual Prompts (Image Overlays)** + - VLMs perform significantly better when their attention is grounded by visual markers. If possible, preprocess screenshots to add bounding boxes or numbered circles over key UI regions, then prompt the model to describe the contents of "Region 1", "Box 2", etc. +3. **Contextual Grounding & Negative Constraints** + - Explicitly instruct the model to avoid guessing: *"You are a precise UI analyzer. Only describe elements that are physically visible in the screenshot. Do not infer hidden functionality, suggest missing buttons, or guess what dropdowns contain."* +4. **Chain-of-Thought (CoT) Visual Reasoning** + - Ask the model to spatially scan the image before answering: *"First, list the distinct sections of the UI from top to bottom. Then, detail the contents of each section."* +5. **Few-Shot Examples** + - Since Qwen-VL supports few-shot learning, providing one example of an image-answer pair demonstrating the exact desired JSON or markdown structure establishes the expected pattern and dramatically reduces hallucinations. + +--- + +## Part 2: OpenClaw Companion App Configuration Analysis + +Based on the provided vision-extracted screenshot (`/tmp/screenshot_desc.md`) and the local architecture documentation (`docs/OPENCLAW_ARCHITECTURE.md`), the OpenClaw macOS companion app is configured as a **Remote Tool Node** connecting to a central Hub. + +Here is the exact required configuration for this MacBook (`100.64.0.12`): + +### 1. Connection & Transport (The "General" Tab) +* **OpenClaw Status**: Active (Checked) +* **Target Mode**: `Remote (another host)` +* **Transport Protocol**: `SSH tunnel` (Must be selected instead of Direct ws/wss). + * *Reasoning:* OpenClaw's security policy prohibits sending operator tokens over unencrypted websockets to remote IPs. The SSH tunnel wraps the traffic and maps it to a secure local loopback (`127.0.0.1:18789`). + +### 2. Network Targeting & Credentials +* **SSH Target**: `root@100.64.0.39:2222` + * *Reasoning:* `100.64.0.39` is the Tailscale mesh IP of the "Luca" server. Port `2222` bypasses the host and connects directly to the SSH daemon running *inside* the `agent-zero` Docker container. +* **Gateway Token**: Must be populated with the shared token. When correctly configured, the UI will display a green `✓ Connected with gateway token` status. The control channel will show `Auth: shared token (gateway/remote.token)`. + +### 3. Feature Approvals (Permissions) +As a tool node, the MacBook requires deep macOS integrations. The following feature toggles must be **Enabled (Checked)** in the UI: +* **Allow Canvas**: For displaying agent outputs. +* **Allow Camera**: For agent visual inputs. +* **Enable Peekaboo Bridge**: *Critical* for allowing signed tools (like 'peekaboo') to drive native UI automation on the Mac. +* **Enable debug tools**: Essential for local troubleshooting. + +### 4. System-Level Prerequisites +For the configuration above to function, the Mac operating system must grant specific permissions to the underlying Node.js binary (as noted in `OPENCLAW_ARCHITECTURE.md`): +* Accessibility +* Screen Recording +* Full Disk Access + +### Summary +The screenshot analyzed by the remote Qwen model perfectly matches the intended architecture. The companion app successfully establishes an encrypted SSH tunnel over Tailscale into the Docker-isolated Hub on Luca, validating its token and registering the Mac as an active node with full Peekaboo UI automation capabilities. \ No newline at end of file diff --git a/README.md b/README.md index 17c2bfd..aee71d7 100644 --- a/README.md +++ b/README.md @@ -57,6 +57,14 @@ This repository documents the OpenClaw gateway integration for the satware AG Ma } ``` +### Companion App UI Configuration +When running via the macOS Companion App, the following settings in the "General" tab are required: +- **Target Mode**: `Remote (another host)` +- **Transport Protocol**: `SSH tunnel` +- **SSH Target**: `root@100.64.0.39:2222` +- **Gateway Token**: Requires the shared gateway token +- **Feature Approvals**: Must enable **Allow Canvas**, **Allow Camera**, **Enable Peekaboo Bridge** (critical for native UI automation), and **Enable debug tools**. + ### Security Policy (`~/.openclaw/exec-approvals.json`) ```json { diff --git a/docs/OPENCLAW_ARCHITECTURE.md b/docs/OPENCLAW_ARCHITECTURE.md index 51b14a0..38aa2ad 100644 --- a/docs/OPENCLAW_ARCHITECTURE.md +++ b/docs/OPENCLAW_ARCHITECTURE.md @@ -47,3 +47,29 @@ The MacBook's node configures the gateway URL as `ws://127.0.0.1:18789` because ### 5. Maintenance Delegation Note: Specific maintenance and configuration tasks associated with this Mac node—including credentials rotation for tools running here, app updates (`OpenClaw.app`), tunnel configuration, and Mac-based logs (`newsyslog`)—are explicitly delegated to this project repository (`mw-macbook-pro`) and the remote operator logged in via `ssh MWsatwareAG@100.64.0.12`. These responsibilities have been intentionally removed from the Manjaro server's maintenance purview to ensure proper separation of concerns. + +### 6. Companion App Configuration (UI Settings) + +The macOS OpenClaw Companion App must be configured with specific settings in the "General" tab: + +* **Target Mode**: `Remote (another host)` +* **Transport Protocol**: `SSH tunnel` +* **SSH Target**: `root@100.64.0.39:2222` (Tailscale IP of Luca, port 2222 bypassing host into Docker) +* **Gateway Token**: Must be populated with the shared token. + +**Feature Approvals**: +The following UI toggles must be **Enabled** for full capability: +* **Allow Canvas**: For displaying agent outputs. +* **Allow Camera**: For agent visual inputs. +* **Enable Peekaboo Bridge**: Critical for native UI automation and vision extraction. +* **Enable debug tools**: Essential for local troubleshooting. + +### 7. VLM Prompting Strategy (Qwen 3.5 9B / Qwen-VL) + +When utilizing Vision Language Models (VLMs) like Qwen 3.5 9B via the gateway to analyze Mac UIs (e.g., screenshots gathered from Peekaboo), the following strategies are required to mitigate hallucinations: + +1. **Structured, Specificity-First Prompting**: Request explicit, format-constrained outputs (e.g., "List every interactive element with its Type, Visible Text, and State") rather than generic descriptions. +2. **Visual Prompts (Image Overlays)**: Ground the VLM's attention using bounding boxes or numbered circles over key UI regions before inference. +3. **Contextual Grounding & Negative Constraints**: Explicitly instruct the model to avoid guessing or inferring hidden functionality. Model must only describe physically visible elements. +4. **Chain-of-Thought (CoT) Visual Reasoning**: Require the model to spatially scan the image (e.g., top to bottom) before answering. +5. **Few-Shot Examples**: Provide example image-answer pairs demonstrating the exact desired structure to establish patterns and dramatically reduce hallucinated elements.