infrastructure/android-fs5
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
aaf008c1bf5f0151978bc35826d82f59388fa83b
android-fs5/specs/002-remote-control/spec.md
T
Jane Alesi aaf008c1bf feat(remote-control): add screen/screenshot/tap/swipe/type/key/launch/ocr/sms commands 
- lib/input.sh: ADB input automation library (screenshot, tap, swipe, type, key, launch, OCR, SMS/OTP via KDE Connect)
- fs5: 9 new remote control commands for AI agent capabilities
- tests/test_remote_control.bats: 31 tests, all passing
- specs/002-remote-control/spec.md: feature specification

Tools: scrcpy 3.3.4 (GUI mirror), tesseract-ocr (OCR), KDE Connect (SMS/OTP)
Constraints: no root required, Android 9 (SDK 28)

Tests: 64/64 passing (33 device mgmt + 31 remote control)
2026-03-02 16:10:11 +01:00

4.7 KiB
Raw Blame History

Feature Specification: Android FS5 Full Remote Control

Feature Branch: 002-remote-control Created: 2026-03-02 Status: Draft Depends on: 001-android-fs5-management (fs5 CLI)

Overview

Extend the FS5 device management to enable full remote control as an AI capability extension: GUI mirroring/automation via scrcpy, SMS/2FA OTP reading via KDE Connect, and TOTP code extraction for automated authentication workflows.

Key insight: KDE Connect is already installed on the device — this is the primary SMS forwarding channel. scrcpy 3.3.4 is already on the host.

User Scenarios & Testing

User Story 1 — GUI Mirroring & Input (Priority: P1)

Why: AI agent needs to see and interact with the phone screen for 2FA app usage.

Acceptance Scenarios:

  1. Given the FS5 is connected, When I run ./fs5 screen, Then scrcpy launches showing the device screen with input forwarding enabled.

  2. When I run ./fs5 screen --headless, Then scrcpy runs without display (for screenshot/OCR automation only).

  3. When I run ./fs5 screenshot, Then a PNG screenshot is saved locally and the path is printed.

  4. When I run ./fs5 tap <x> <y>, Then a tap event is sent to the device at those coordinates.

  5. When I run ./fs5 swipe <x1> <y1> <x2> <y2>, Then a swipe gesture is performed on the device.

  6. When I run ./fs5 type "text", Then the text is typed into the currently focused input field.

User Story 2 — SMS/OTP Reading via KDE Connect (Priority: P1)

Why: Receive SMS 2FA codes without manual phone interaction.

Acceptance Scenarios:

  1. When I run ./fs5 sms list, Then recent SMS messages are listed with sender, timestamp, and body.

  2. When I run ./fs5 sms watch, Then new SMS messages are printed as they arrive (polling loop).

  3. When I run ./fs5 sms otp, Then the most recent OTP/2FA code (4-8 digit number) is extracted and printed.

  4. When I run ./fs5 sms otp --wait, Then the tool waits up to 60 seconds for a new OTP SMS and prints the code.

User Story 3 — TOTP Code Reading from Authenticator App (Priority: P2)

Why: AI agent needs TOTP codes from apps like Google Authenticator or andOTP.

Acceptance Scenarios:

  1. When I run ./fs5 totp screenshot, Then a screenshot of the authenticator app is taken and the current TOTP code is extracted via OCR.

  2. When I run ./fs5 totp read --app <package>, Then the authenticator app is opened, a screenshot taken, and the 6-digit code extracted.

User Story 4 — ADB Input Automation (Priority: P2)

Why: Automate button presses, app launches, and navigation for 2FA flows.

Acceptance Scenarios:

  1. When I run ./fs5 key <keycode>, Then the Android key event is sent (e.g., KEYCODE_HOME, KEYCODE_BACK).

  2. When I run ./fs5 launch <package>, Then the specified app is launched on the device.

  3. When I run ./fs5 ocr, Then a screenshot is taken and all visible text is extracted via OCR.

Requirements

Functional Requirements

  • FR-010: screen command MUST launch scrcpy with device serial targeting.
  • FR-011: screenshot MUST save PNG to configurable local path.
  • FR-012: tap, swipe, type MUST use adb shell input commands.
  • FR-013: sms list MUST read SMS via KDE Connect CLI or ADB broadcast.
  • FR-014: sms otp MUST extract 4-8 digit codes from SMS body via regex.
  • FR-015: sms watch MUST poll for new SMS at configurable interval.
  • FR-016: totp MUST use screenshot + OCR (tesseract) for code extraction.
  • FR-017: All new commands MUST support --json output.

Non-Functional Requirements

  • NFR-010: Screenshot capture MUST complete in < 5 seconds.
  • NFR-011: OTP extraction MUST complete in < 3 seconds after SMS arrives.
  • NFR-012: OCR MUST achieve > 95% accuracy on 6-digit numeric codes.
  • NFR-013: No root required on device.

Tool Selection (Research-Based)

Tool Purpose Status Rationale
scrcpy 3.3.4 GUI mirror + input ✅ Installed Best no-root screen control
KDE Connect SMS forwarding ✅ On device Already installed, native Linux integration
adb shell input Tap/swipe/type ✅ Built-in No extra deps
tesseract-ocr TOTP OCR Install needed Best open-source OCR
kdeconnect-cli SMS reading Install needed CLI for KDE Connect

Out of Scope

  • Wireless ADB setup (USB connection sufficient)
  • Rooted device operations
  • Full Appium test framework (overkill for this use case)
  • SMS sending (receive only)
Reference in New Issue View Git Blame Copy Permalink