OpenClaw macOS Node Configuration & VLM Prompting Research

Part 1: VLM Prompting Strategy to Mitigate Hallucinations (Qwen 3.5 9B / Qwen-VL)

Based on recent deep research into Vision Language Models (VLMs) and how to prompt them for dense visual interfaces like UI screenshots, the following techniques are required to extract accurate, hallucination-free results from models like Qwen 3.5 9B:

Structured, Specificity-First Prompting
- Do not use generic prompts like "Describe this UI".
- Do ask for explicit, format-constrained outputs: "List every interactive element in this UI. For each element, provide: [Type (e.g., button, input, toggle)], [Visible Text], [State (e.g., active, grayed out)]."
Visual Prompts (Image Overlays)
- VLMs perform significantly better when their attention is grounded by visual markers. If possible, preprocess screenshots to add bounding boxes or numbered circles over key UI regions, then prompt the model to describe the contents of "Region 1", "Box 2", etc.
Contextual Grounding & Negative Constraints
- Explicitly instruct the model to avoid guessing: "You are a precise UI analyzer. Only describe elements that are physically visible in the screenshot. Do not infer hidden functionality, suggest missing buttons, or guess what dropdowns contain."
Chain-of-Thought (CoT) Visual Reasoning
- Ask the model to spatially scan the image before answering: "First, list the distinct sections of the UI from top to bottom. Then, detail the contents of each section."
Few-Shot Examples
- Since Qwen-VL supports few-shot learning, providing one example of an image-answer pair demonstrating the exact desired JSON or markdown structure establishes the expected pattern and dramatically reduces hallucinations.

Part 2: OpenClaw Companion App Configuration Analysis

Based on the provided vision-extracted screenshot (/tmp/screenshot_desc.md) and the local architecture documentation (docs/OPENCLAW_ARCHITECTURE.md), the OpenClaw macOS companion app is configured as a Remote Tool Node connecting to a central Hub.

Here is the exact required configuration for this MacBook (100.64.0.12):

1. Connection & Transport (The "General" Tab)

OpenClaw Status: Active (Checked)
Target Mode: Remote (another host)
Transport Protocol: SSH tunnel (Must be selected instead of Direct ws/wss).
- Reasoning: OpenClaw's security policy prohibits sending operator tokens over unencrypted websockets to remote IPs. The SSH tunnel wraps the traffic and maps it to a secure local loopback (127.0.0.1:18789).

2. Network Targeting & Credentials

SSH Target: root@100.64.0.39:2222
- Reasoning: 100.64.0.39 is the Tailscale mesh IP of the "Luca" server. Port 2222 bypasses the host and connects directly to the SSH daemon running inside the agent-zero Docker container.
Gateway Token: Must be populated with the shared token. When correctly configured, the UI will display a green ✓ Connected with gateway token status. The control channel will show Auth: shared token (gateway/remote.token).

3. Feature Approvals (Permissions)

As a tool node, the MacBook requires deep macOS integrations. The following feature toggles must be Enabled (Checked) in the UI:

Allow Canvas: For displaying agent outputs.
Allow Camera: For agent visual inputs.
Enable Peekaboo Bridge: Critical for allowing signed tools (like 'peekaboo') to drive native UI automation on the Mac.
Enable debug tools: Essential for local troubleshooting.

4. System-Level Prerequisites

For the configuration above to function, the Mac operating system must grant specific permissions to the underlying Node.js binary (as noted in OPENCLAW_ARCHITECTURE.md):

Accessibility
Screen Recording
Full Disk Access

Summary

The screenshot analyzed by the remote Qwen model perfectly matches the intended architecture. The companion app successfully establishes an encrypted SSH tunnel over Tailscale into the Docker-isolated Hub on Luca, validating its token and registering the Mac as an active node with full Peekaboo UI automation capabilities.

4.1 KiB Raw Blame History