How it works

Curious about how AutoPrompt can transform your workday? This page breaks down the magic behind our automation assistant, showing you how it enables LLMs to control other computers remotely and boost your productivity. Discover how AutoPrompt can become your silent work partner.

How to setup

Take a look at the diagram to see where everything fits in. The HDMI Capture device lets AutoPrompt Companion "see" your target computer screen, while delivering mouse and keyboard commands to the AutoPrompt via Bluetooth. The AutoPrompt transmits those commands to the target computer as standard USB keyboard and mouse inputs. 

AutoPrompt in action

You prompt the LLM. AutoPrompt Companion shows the LLM your target computer's screen. The LLM works out the keyboard and mouse inputs required to realise your prompt and AutoPrompt Companion transmits those inputs via Bluetooth to the AutoPrompt device.

Effortless automation

The name AutoPrompt comes from our software's unique ability to instruct an LLM to break a complex task into multiple sub-tasks, then continuously prompt it to work through those sub-tasks until the main one is complete. Our software has the LLM assess its progress along the way, so that it can correct mistakes as it goes. 

 

Use our custom Switch feature to endlessly prompt an LLM to work on a task. AutoPrompt Companion will work through sub-tasks until it reaches the end, then it will work through those sub-tasks again, continuosly completing the main task until the user deactivates the Switch.

Improved workday

With AutoPrompt , you can train an LLM to complete virtually any task on another computer using carefully crafted prompts. The AutoPrompt is detected as a standard USB mouse and keyboard, requiring no software installation on the other computer. Only your personal computer needs AutoPrompt Companion, making integration seamless and secure.

Settings Menu Explained

Hardware
  • Camera Index: Tells the program which camera to use for screen capture. Set this to “0” for your main camera or “1” if you’re using a second one (like an HDMI capture card).
  • Bluetooth Baud: The communication speed between your phone/computer and the AutoPrompt device. Keep it at 9600 unless you’ve changed your device’s settings.
  • BT Timeout (s): How long to wait (in seconds) for a response from the AutoPrompt device before giving up. 1.0 second works for most setups.

AI
  • OpenRouter API Key: Your personal key for accessing AI models. You’ll get this from OpenRouter.ai.
  • Model: Choose which AI brain to use (like GPT-4o or Claude). You can also add your own custom models.
  • Add Model: Type the name of a new model you’d like to use and click “Add” to save it for future use.
  • System Prompt: The core instructions that tell the AI how to behave, what commands to use, and how to respond. You can tweak this to change the AI’s style or add new rules—but keep the technical command formats intact. See next section for detailed information.

Behaviour
  • Key Delay Sec: How long (in seconds) to wait between each keystroke. Increase this if your computer seems to miss keys.
  • System Combo Delay Sec: Extra pause after keyboard shortcuts like Win+R or Ctrl+S—useful for slower apps.
  • Mouse Move Delay Sec: Small pause between mouse movements to mimic natural motion.
  • Click Delay Sec: Pause after mouse clicks to ensure they register.
  • Max Actions Per Turn: Limits how many commands the AI can send in one prompt. Useful if you want a hard limit to how much the AI can do.
  • Enable Mouse: Turn mouse control on or off. Disable this if you only want keyboard automation.
  • Auto Prompt Iterations: How many steps the AI should plan ahead during Auto-Prompt mode.
  • Auto Prompt Mode: “Fixed” runs your plan exactly as written; “until_complete” lets the AI keep going until it sees the goal is done.
  • Show Debug Console: Turns on a detailed log window for troubleshooting. Helpful for testing, but not needed for regular use.

Safety
  • Confirm before executing AI commands: If turned on, you’ll get a popup asking you to approve every batch of commands before they run. Great for cautious use or testing new prompts.

Stealth
  • Randomize Delays (±20%): Adds slight, natural-looking variations to your delays.
  • Inject Mistakes (typos + corrections): Occasionally makes and fixes human-like typos (like typing “c” instead of “v” then backspacing). Helps your automation appear more human. All typos are handled by the program, rather than the LLM, and will ALWAYS be corrected.

Custom Controls
  • Buttons (4 total): Create one-click actions! Give each a name and a prompt. Turn on “Auto-Prompt” if you want it to run a full multi-step plan.
  • Switches (2 total): These toggle on/off endless automation loops—perfect for “look busy” modes or background tasks. Unlike buttons, but they keep running until you turn them off.

System Prompt Explained

The system prompt is the core instruction manual that guides your AI assistant on how to interact with your other computer. It explains that the AI’s job is to control your other PC using only keyboard and mouse commands, based on what it sees in the live video feed. The prompt encourages the AI to work step by step: first observe what's on screen, then decide on the next small action, carry it out using the correct command format, and always double-check that the action worked before moving on. This careful, visual approach helps ensure reliable and accurate automation.

Default System Prompt

You are a Windows 10/11 automation assistant with vision. Your task is to control a target computer exclusively via mouse and keyboard inputs, based on instructions and the live camera feed you receive.

Always follow this workflow, first observe the current screen state then plan the next immediate action, next execute using only valid HID commands, and finally never assume the goal is complete because the commands were sent, always verify via the image.

### KEYBOARD COMMANDS
Format: "k:modifier,usage_id" — exactly two numbers separated by a comma.
Modifier values: 0=none, 1=Ctrl, 2=Shift, 4=Alt, 8=Win
Usage IDs: a=4, b=5, c=6, d=7, e=8, f=9, g=10, h=11, i=12, j=13, k=14, l=15, m=16, n=17, o=18, p=19, q=20, r=21, s=22, t=23, u=24, v=25, w=26, x=27, y=28, z=29, 1=30, 2=31, 3=32, 4=33, 5=34, 6=35, 7=36, 8=37, 9=38, 0=39, Enter=40, Backspace=42, Tab=43, Space=44, F1=58, F2=59, ..., F12=69, Up=82, Down=81, Left=80, Right=79, Home=83, End=84.
✅ Examples: 'a' = "k:0,4", Enter = "k:0,40", Win+R = "k:8,21", Ctrl+S = "k:1,22"
❌ NEVER use: "k:4", "k:a", or single-number formats.

### MOUSE COMMANDS
All movements are RELATIVE to current cursor position.
- Move: "m:dx,dy" (dx/dy ∈ [-50, 50])
- Click: "c:left", "c:right"
- Scroll: "s:units"
⚠️ Do NOT use absolute coordinates (e.g., m:500,300). Break large moves into small steps.

### COMMON TASKS
- Delete line: Up/Down to line → "k:2,84" (Shift+End) → "k:0,42" (Backspace)
- Prefer keyboard shortcuts (Alt+F4, Win+R) over mouse when possible

### OUTPUT FORMAT
ALWAYS reply with JSON: {"thought": "...", "actions": ["k:0,40", "m:20,-10"]}
Actions must be STRINGS — no objects like {"k": "0,40"}.
In 'thought', briefly state your reasoning.

If you’d like to customize the system prompt, there are a few key parts worth keeping to maintain smooth communication with the program. These include the exact keyboard command format (k:modifier,usage_id, like k:0,4 for the letter "a"), the mouse command formats (m:dx,dy for movement, c:left for clicks, etc.), and the instruction to always reply in structured JSON like {"thought": "...", "actions": ["k:0,40", "m:10,5"]}. These elements act as the common language between the AI and your AutoPrompt Companion, and it’s the same language used by a standard USB mouse and keyboard right off the store shelf. Keeping this in your system prompt will help everything run smoothly while still giving you room to adjust tone, add examples, or refine task instructions as needed.