Jane Alesi
aaf008c1bf
- lib/input.sh: ADB input automation library (screenshot, tap, swipe, type, key, launch, OCR, SMS/OTP via KDE Connect) - fs5: 9 new remote control commands for AI agent capabilities - tests/test_remote_control.bats: 31 tests, all passing - specs/002-remote-control/spec.md: feature specification Tools: scrcpy 3.3.4 (GUI mirror), tesseract-ocr (OCR), KDE Connect (SMS/OTP) Constraints: no root required, Android 9 (SDK 28) Tests: 64/64 passing (33 device mgmt + 31 remote control)
138 lines
4.7 KiB
Markdown
138 lines
4.7 KiB
Markdown
# Feature Specification: Android FS5 Full Remote Control
|
|
|
|
**Feature Branch**: `002-remote-control`
|
|
**Created**: 2026-03-02
|
|
**Status**: Draft
|
|
**Depends on**: 001-android-fs5-management (fs5 CLI)
|
|
|
|
---
|
|
|
|
## Overview
|
|
|
|
Extend the FS5 device management to enable **full remote control** as an AI capability
|
|
extension: GUI mirroring/automation via scrcpy, SMS/2FA OTP reading via KDE Connect,
|
|
and TOTP code extraction for automated authentication workflows.
|
|
|
|
**Key insight**: KDE Connect is already installed on the device — this is the primary
|
|
SMS forwarding channel. scrcpy 3.3.4 is already on the host.
|
|
|
|
---
|
|
|
|
## User Scenarios & Testing
|
|
|
|
### User Story 1 — GUI Mirroring & Input (Priority: P1)
|
|
|
|
**Why**: AI agent needs to see and interact with the phone screen for 2FA app usage.
|
|
|
|
**Acceptance Scenarios**:
|
|
|
|
1. **Given** the FS5 is connected, **When** I run `./fs5 screen`,
|
|
**Then** scrcpy launches showing the device screen with input forwarding enabled.
|
|
|
|
2. **When** I run `./fs5 screen --headless`,
|
|
**Then** scrcpy runs without display (for screenshot/OCR automation only).
|
|
|
|
3. **When** I run `./fs5 screenshot`,
|
|
**Then** a PNG screenshot is saved locally and the path is printed.
|
|
|
|
4. **When** I run `./fs5 tap <x> <y>`,
|
|
**Then** a tap event is sent to the device at those coordinates.
|
|
|
|
5. **When** I run `./fs5 swipe <x1> <y1> <x2> <y2>`,
|
|
**Then** a swipe gesture is performed on the device.
|
|
|
|
6. **When** I run `./fs5 type "text"`,
|
|
**Then** the text is typed into the currently focused input field.
|
|
|
|
---
|
|
|
|
### User Story 2 — SMS/OTP Reading via KDE Connect (Priority: P1)
|
|
|
|
**Why**: Receive SMS 2FA codes without manual phone interaction.
|
|
|
|
**Acceptance Scenarios**:
|
|
|
|
1. **When** I run `./fs5 sms list`,
|
|
**Then** recent SMS messages are listed with sender, timestamp, and body.
|
|
|
|
2. **When** I run `./fs5 sms watch`,
|
|
**Then** new SMS messages are printed as they arrive (polling loop).
|
|
|
|
3. **When** I run `./fs5 sms otp`,
|
|
**Then** the most recent OTP/2FA code (4-8 digit number) is extracted and printed.
|
|
|
|
4. **When** I run `./fs5 sms otp --wait`,
|
|
**Then** the tool waits up to 60 seconds for a new OTP SMS and prints the code.
|
|
|
|
---
|
|
|
|
### User Story 3 — TOTP Code Reading from Authenticator App (Priority: P2)
|
|
|
|
**Why**: AI agent needs TOTP codes from apps like Google Authenticator or andOTP.
|
|
|
|
**Acceptance Scenarios**:
|
|
|
|
1. **When** I run `./fs5 totp screenshot`,
|
|
**Then** a screenshot of the authenticator app is taken and the current TOTP code is extracted via OCR.
|
|
|
|
2. **When** I run `./fs5 totp read --app <package>`,
|
|
**Then** the authenticator app is opened, a screenshot taken, and the 6-digit code extracted.
|
|
|
|
---
|
|
|
|
### User Story 4 — ADB Input Automation (Priority: P2)
|
|
|
|
**Why**: Automate button presses, app launches, and navigation for 2FA flows.
|
|
|
|
**Acceptance Scenarios**:
|
|
|
|
1. **When** I run `./fs5 key <keycode>`,
|
|
**Then** the Android key event is sent (e.g., KEYCODE_HOME, KEYCODE_BACK).
|
|
|
|
2. **When** I run `./fs5 launch <package>`,
|
|
**Then** the specified app is launched on the device.
|
|
|
|
3. **When** I run `./fs5 ocr`,
|
|
**Then** a screenshot is taken and all visible text is extracted via OCR.
|
|
|
|
---
|
|
|
|
## Requirements
|
|
|
|
### Functional Requirements
|
|
|
|
- **FR-010**: `screen` command MUST launch scrcpy with device serial targeting.
|
|
- **FR-011**: `screenshot` MUST save PNG to configurable local path.
|
|
- **FR-012**: `tap`, `swipe`, `type` MUST use `adb shell input` commands.
|
|
- **FR-013**: `sms list` MUST read SMS via KDE Connect CLI or ADB broadcast.
|
|
- **FR-014**: `sms otp` MUST extract 4-8 digit codes from SMS body via regex.
|
|
- **FR-015**: `sms watch` MUST poll for new SMS at configurable interval.
|
|
- **FR-016**: `totp` MUST use screenshot + OCR (tesseract) for code extraction.
|
|
- **FR-017**: All new commands MUST support `--json` output.
|
|
|
|
### Non-Functional Requirements
|
|
|
|
- **NFR-010**: Screenshot capture MUST complete in < 5 seconds.
|
|
- **NFR-011**: OTP extraction MUST complete in < 3 seconds after SMS arrives.
|
|
- **NFR-012**: OCR MUST achieve > 95% accuracy on 6-digit numeric codes.
|
|
- **NFR-013**: No root required on device.
|
|
|
|
---
|
|
|
|
## Tool Selection (Research-Based)
|
|
|
|
| Tool | Purpose | Status | Rationale |
|
|
|------|---------|--------|-----------|
|
|
| scrcpy 3.3.4 | GUI mirror + input | ✅ Installed | Best no-root screen control |
|
|
| KDE Connect | SMS forwarding | ✅ On device | Already installed, native Linux integration |
|
|
| adb shell input | Tap/swipe/type | ✅ Built-in | No extra deps |
|
|
| tesseract-ocr | TOTP OCR | Install needed | Best open-source OCR |
|
|
| kdeconnect-cli | SMS reading | Install needed | CLI for KDE Connect |
|
|
|
|
## Out of Scope
|
|
|
|
- Wireless ADB setup (USB connection sufficient)
|
|
- Rooted device operations
|
|
- Full Appium test framework (overkill for this use case)
|
|
- SMS sending (receive only)
|