Automation Skill Builder

User manual (customers) · local web app + API + optional MCP · Database MCP build & MCP tools (troubleshooting)

1. Overview

This document is the customer-facing manual: setup, settings, everyday use, and troubleshooting. Packaging, billing servers, and engineering internals ship with the downloadable source package; they are not published on this website.

The product is a FastAPI-based automation and skill-building workstation. It combines a browser UI, desktop helpers (screenshots, OCR, mouse/keyboard), optional Playwright browser automation (MCP relay), FastSkills-oriented skill packaging, and an optional MCP server for AI assistants.

Who this manual is for

• Everyday users — install, open the browser UI, run automations without using a terminal.
• Integrators — connect schedulers, agents, or MCP clients to local HTTP/MCP endpoints.
• Operators — platform matrix, environment variables, quota, and troubleshooting (§2, §8).

Suggested reading paths

• First day: Quick start → First capture tutorial → FAQ.
• Connect an AI assistant: Settings → HTTP API & MCP → Database MCP prompts.
• Deployment planning: Platform requirements → Component matrix.

Quick start (about 5 minutes)

Follow these steps after you install the app for your OS. For prerequisites, see §2.1.

Step 1: Start the application

1. Launch Automation Skill Builder from your Windows .exe, macOS .app, or Linux launcher.
2. Allow any system permission prompts (especially on macOS — see §8.1).
3. Wait until the tray icon or splash indicates the local service is running.

Step 2: Open the control panel in your browser

1. Open a current Chrome or Edge (Chromium) window.
2. Go to http://127.0.0.1:8800 (or the port you configured).
3. Confirm the sidebar loads (script editor, utilities, Settings).

Step 3: Set your language and verify capabilities

1. Open Settings → General and choose your display language.
2. Click Save in the header if prompted.
3. Open MCP & server utilities from the sidebar → Advanced to review FastSkills, Playwright, and optional MCP status.

Step 4: Run your first action

1. From the sidebar, open a utility such as Screenshot or follow the first capture tutorial below.
2. Confirm the result appears in the UI or in your configured output folder.

Next steps: Common scenarios · Settings & security · FAQ

2. Installation & environment

2.1 Requirements

• Packaged app: use the build for your OS; you do not need Python installed just to run the distributed executable. Skill execution and some automations may still need Python and Node.js as described on the product download page and in Settings / the in-app capability checklist. Browser automation prerequisites are summarized in the matrix below; technical browser setup is documented in the application source package for licensed developers.
• From source (developers): Python 3.10+ and the dependency files that ship with the source package — see the README and dependency files in the application repository for virtualenv setup.
• For PaddleOCR / vision features, expect large downloads on first run (model cache under the user profile, e.g. .paddlex).

Platform requirements (Windows, macOS, Linux)

The product documentation does not name a single minimum OS build for every deployment (e.g. “Windows 10 1903” or “macOS 12”). What follows reflects practical use of the ML/desktop stack together with Python and vendor requirements for your machine.

Practical minimums (all platforms)

• Architecture: 64-bit OS recommended (wheels for Paddle, PyTorch, and related stacks target 64-bit).
• RAM: 8 GB+ recommended for a comfortable full install (OCR, Docling, browser automation, UI). 4 GB is unlikely to be reliable for the complete dependency set.
• Disk: several gigabytes beyond the product install for dependencies (if you use a full stack) plus on-demand model caches (e.g. under .paddlex).
• Browser: a current Chromium-based desktop browser for the web UI over HTTP(S) (default port is often 8800). The UI expects roughly Chrome / Edge (Chromium) 80+ (modern JavaScript). Legacy systems: see §8.6.
• Network: needed at least for first-time model downloads and optional integrations (PyPI, uvx, upstream AI APIs unless you pre-cache artifacts).

Windows

• OS: use 64-bit Windows 10 or later (or current Windows Server) that supports your chosen Python build and vendor wheels when you rely on Python-based features.
• Dependencies: install platform extras from requirements-windows.txt (includes pywin32 >= 306, PyAutoGUI, pynput, and other Windows-oriented packages listed there). Project builds exclude macOS-only pyobjc on Windows.
• WSL: if you run Python inside WSL, treat the environment as Linux for dependencies and permissions—not as native Windows.

macOS

• OS: use a recent macOS release that supports Python 3.10+ and the dependency stack you install (python.org or supported tooling). Download or support materials for your build list concrete prerequisites.
• Dependencies: full macOS installs from source include pyobjc and related packages not used on Linux.
• Permissions (System Settings → Privacy & Security): desktop automation needs the right entries enabled for the Skill Builder app (and for Terminal or your IDE if you start the server from there). Typical items:
  • Accessibility — mouse, keyboard, and many automation actions.
  • Input Monitoring — may be required in addition to Accessibility for some input paths.
  • Screen Recording — often required for screenshots and screen-based OCR flows.
  • Automation (control of other apps) — may appear when the product drives other applications.
  macOS usually shows a prompt the first time; you can also add or toggle the app manually in each list. After changing permissions, quit and restart Skill Builder (and sometimes Terminal/IDE) so the new grants take effect—see also §8.1.

Linux

• Display: the optional tray/splash GUI expects a graphical session (DISPLAY on X11/Wayland setups). Use --no-gui for headless servers when you only need HTTP.
• Host / remote APIs (window_title): on Linux the app does not use PyGetWindow (Windows-oriented). Install xdotool (sudo apt install xdotool) so responses can include the active window title; pure Wayland sessions may still return Unknown.

2.1.5 Component availability matrix (base vs optional)

The matrix below lists a practical bottom line per OS column (what must be true on the host for that row to work). It is not a vendor certification—exact build numbers follow Python, Node/Playwright, Docker, and uvx documentation for your machine. Docker adds host rules from the Docker vendor (RAM, Windows WSL 2 / Hyper-V, minimum macOS / Linux kernel). If Database MCP uses Docker at runtime, those apply on top of the Skill Builder baseline. The Linux column assumes typical 64-bit glibc distributions (e.g. Debian/Ubuntu family); other distros may need extra work for ML wheels. Linux dependency lockfiles for source installs are documented in the application source package.

Legend: rows are major stacks; columns are host OS families. “Optional” integrations can be disabled in their config files if you do not need them.

Avoid a “double dead-end” on old hardware (example: pre-2015 Mac notebooks): You might install the core Python app, then discover that Database MCP has no usable native db-mcp-server binary in your bundle or build, so you plan to rely on the Docker fallback—only to find that Docker Desktop no longer supports your macOS version (or the machine lacks RAM / virtualisation support). Docker is not a universal escape hatch. Before counting on Database MCP, check Docker’s current host requirements for your exact OS build; on very old Macs, plan to run Database MCP on a newer host, use a pre-built native binary from a supported release channel, or leave Database MCP disabled if you only need the rest of the app.

The in-app Settings → Advanced → Capabilities report shows live status for FastSkills, Playwright MCP, Database MCP, and Python Interpreter MCP. Offline / air-gapped notes: §8.5 (FastSkills). Python Interpreter MCP (stdio, uvx, external mode): documented in the application source package.

2.2 Environment variables (optional)

• AUTOMATION_SERVICE_HOST / AUTOMATION_SERVICE_PORT — bind address and port (default port often 8800).
• AUTOMATION_SKIP_OCR_WARMUP — skip eager Paddle warmup at startup if set appropriately (see app hints).
• AUTOMATION_CORS_ORIGINS — CORS when the web UI is served from a different origin than the API.

Packaged desktop builds may use different launch flags. Operators and developers: additional environment variables (billing, recording, MCP) are documented in the application repository.

3. Running the application

Start the product with the installed executable or launcher from your platform’s package (Windows .exe, macOS .app, or your Linux distribution’s start method). The local web UI and API listen on the configured port (often 8800 for HTTP).

Open http://127.0.0.1:8800 (or your chosen host/port) in a browser. API docs are typically at /docs and OpenAPI at /openapi.json.

By default, the full app also serves HTTPS on port 8843 (see AUTOMATION_HTTPS_PORT). To disable HTTPS, use your build’s documented launch flags or set AUTOMATION_USE_HTTPS=0 if supported. Running from a Python source checkout (python main.py / uvicorn) is documented in the application source package.

Port conflicts: If 8800 is taken (e.g. by another stack), set AUTOMATION_SERVICE_PORT or pass --port to the launcher.

3.1 HTTPS & trusting the certificate (Chrome)

The first time HTTPS is used, the app creates a self-signed certificate under a tls/ folder next to your app data root (beside the .exe on Windows; inside .app/Contents/MacOS/ on macOS; or next to the runtime root for your install). For a source checkout, paths follow the project layout — see the application source package. Files: tls/localhost.crt (public) and tls/localhost.key (private). The console prints the full paths when the cert is generated or replaced.

The certificate includes SAN entries for localhost, open-skills.local, 127.0.0.1, and ::1. If you use open-skills.local, add 127.0.0.1 open-skills.local to your hosts file. After the app regenerates the cert (e.g. to add a new name), trust the new localhost.crt again.

Chrome on macOS — remove the “Not secure” / certificate warning

1. Open Keychain Access (or double-click localhost.crt).
2. Import localhost.crt if needed, then select the certificate (e.g. “Automation Studio Local”).
3. Open Trust → set When using this certificate to Always Trust (at least for Secure Sockets Layer (SSL)).
4. Quit Chrome completely (Chrome → Quit Google Chrome or Cmd+Q), then reopen — closing tabs alone is not enough.
5. Visit https://127.0.0.1:8843/docs or https://open-skills.local:8843/docs; Chrome should treat the site as trusted.

Windows (brief): Import localhost.crt into Trusted Root Certification Authorities for the current user (e.g. via Certificate Manager / certmgr.msc), then fully restart Chrome.

Install only the .crt file into the trust store. Never share the .key file. Self-signed local certs are for development; do not use them for production internet-facing HTTPS.

4. Main UI & navigation

• Sidebar — access to major modes: script editor, utilities, AI-assisted flows, recording, and Settings.
• Script / skill panel — edit Python or workflow-related content; generate packages, ZIPs, or integration with FastSkills where configured.
• Modals — OCR, screenshots, parameter extraction (“AI-assisted code revision”), listener service, find-latest-file, etc., open as overlays.
• Welcome / language — display language can be changed in Settings → General.

5. Settings

Open Settings from the sidebar. Tabs include:

• General — machine ID, app version / update check, UI language, welcome page, browser entry (HTTP/HTTPS), service API key, run-code timeout, Playwright browser profile (isolated / persistent / CDP), etc.
• Subscription & license — license upload/status if enabled in your deployment.
• Request usage — open MCP & server utilities from the sidebar → Usage tab (same counters as monthly quota rules).
• Security — proxy chat allowlist and related options.
• Model selection — endpoints for proxy chat / code analysis providers.
• Credits — third-party component notices.
• Database MCP — optional FreePeak db-mcp-server (connections JSON, tools prefixed asb_db_). See §5.1 for Docker image / SQLite paths; §5.2 for how to phrase tool + SQL requests to an AI assistant.

Many security-sensitive options require Save in the header and sometimes an application restart (on-screen hints apply).

5.1 Database MCP — Docker image (first-time pull)

When no native db-mcp-server binary is present (PyInstaller bundle or vendor/db_mcp_server/), the app can run Database MCP through Docker if the image already exists locally. The app uses docker image inspect to detect that; it does not run docker pull for you.

Runtime: native binary vs Docker vs neither

1) Native db-mcp-server available (bundled next to the frozen app under _internal, or vendor/db_mcp_server/db-mcp-server(.exe) in dev)

• The app spawns that executable for Database MCP stdio. Docker is not required at runtime for this path.
• Advanced: set DB_MCP_SERVER_FORCE_DOCKER=1 to ignore the vendor binary and force the Docker image instead (see the application repository documentation).

2) No native binary — Docker fallback

• docker must be on PATH (GUI-launched apps may not inherit Docker’s PATH; fix the environment or use a full path).
• The image in DB_MCP_SERVER_RUNTIME_IMAGE (default freepeak/db-mcp-server:v1.8.0) must already be pulled: docker pull … (below).
• Set DB_MCP_SERVER_DISABLE_DOCKER=1 to turn off this fallback (only native binary will be considered).

3) Neither native binary nor a pulled Docker image

• Database MCP does not register (no asb_db_* tools). Startup logs print hints such as “no native binary and Docker fallback unavailable”.
• The rest of the application still runs (web UI, other APIs, other MCP tools).
• Fix: pull the image (this section), or add a native binary under vendor/ / rebuild the .exe with the binary bundled, or leave Database MCP disabled in db_mcp_server_settings.json (enabled: false) if you do not need it.

Command line (fastest)

Run in a terminal:

docker pull freepeak/db-mcp-server:v1.8.0

With Docker Desktop, this uses the same engine as the GUI. After a successful pull, the in-app check for the Docker fallback image should pass.

For another tag or a private image, set DB_MCP_SERVER_RUNTIME_IMAGE to match, then pull that reference, for example:

export DB_MCP_SERVER_RUNTIME_IMAGE=freepeak/db-mcp-server:v1.8.0
docker pull "$DB_MCP_SERVER_RUNTIME_IMAGE"

Docker Desktop (GUI)

1. Open Docker Desktop.
2. Open Images.
3. Click Pull (or use Search to find the image).
4. Enter freepeak/db-mcp-server:v1.8.0 and confirm the pull.
5. When finished, the image appears in the Images list.

SQLite on the host while using Docker

When Database MCP starts db-mcp-server in Docker, this application bind-mounts the parent directory of your active connections JSON (the folder that contains db_mcp_server_config.json, or the file pointed to by AUTOMATION_DB_MCP_CONFIG_PATH) read-write at /asb-mcp-data inside the container. In connections, set SQLite to an absolute path under that mount, for example:

"database_path": "/asb-mcp-data/db_mcp_local.sqlite"

The database file then lives on the host next to the config file. To use another host folder instead, set DB_MCP_SERVER_DOCKER_DATA_DIR to that folder’s absolute path before starting the automation server. The whole mounted directory is visible inside the container — only mount directories you are comfortable exposing to the db-mcp image.

If you rely on SQLite with Database MCP inside Docker, the default upstream image tag may not match your needs. Operators should read Database MCP build & troubleshooting.

5.2 Database MCP — asking an AI assistant

With the app’s MCP endpoint connected in your client (for example streamable HTTP at http://127.0.0.1:8800/mcp-http), you do not need to guess JSON field names for tools. Say clearly which tool to invoke and which SQL to run; the model maps that to the tool schema (sql, query, etc.) for you.

Tool names follow the connection id in your connections JSON — for example local_sqlite yields asb_db_query_local_sqlite, asb_db_schema_local_sqlite, and so on. Replace local_sqlite in the examples below if your id differs.

Chinese (copy-paste)

• 「请调用 asb_db_query_local_sqlite，执行 SQL：SELECT sqlite_version();，我要看 SQLite 版本。」
• 「用 Database MCP 的 asb_db_query_local_sqlite 查一下当前库的 SQLite 版本。」

English (copy-paste)

• “Please call asb_db_query_local_sqlite with SQL SELECT sqlite_version(); to show the SQLite version.”

More checks (optional)

• SELECT sqlite_version(); is enough for the library version string.
• For extra engine detail, ask the assistant to run PRAGMA compile_options; or SELECT sqlite_source_id(); via the same query tool — only if that tool allows those statements.

6. Common usage scenarios

Tutorial: Your first screen capture

Time: about 10 minutes · Difficulty: Beginner · Prerequisites: app running, browser open at http://127.0.0.1:8800

What you will do: capture a region of your desktop and confirm the image is saved or shown in the UI.

1. In the sidebar, open the Screenshot (or equivalent capture) utility.
2. Select a screen region, or accept the default full-screen capture if offered.
3. Wait for processing — first OCR-related runs may download models (see §8.2).
4. Verify the preview or file path shown in the dialog.
5. Optional: narrow the capture region next time to speed up OCR workflows (see §6.1).

Pro tip: On macOS, grant Screen Recording if the capture is blank — then quit and restart the app.

6.1 Desktop OCR → find text → click

Use POST /api/desktop/desktop_ocr_action (MCP tool id desktop_ocr_click_text when exposed). You can pass region, wait seconds before capture, target_text, mouse action, and optional offsets. On macOS, mouse execution may use pynput / Quartz before PyAutoGUI (see server implementation).

6.2 Programmatic mouse & keyboard

Use POST /api/host/mouse/move and POST /api/host/mouse/click for mouse automation. The in-app Mouse Click flow records a replay script that calls /api/host/mouse/click with button and clicks (e.g. double-click = left + clicks: 2); omit x/y to click at the cursor position when the request is handled (after any client-side countdown).

Use POST /api/host/keyboard/type, …/press, and …/hotkey for typing and key combinations. Other desktop flows (window focus, region screenshot, launch/close app, etc.) are under /api/host/* in OpenAPI (canonical host automation prefix).

Older URLs such as /api/remote/mouse/click and /api/action/* aliases still work but are hidden from OpenAPI; new integrations should use /api/host/*.

6.3 Browser automation

Playwright is exposed via relay routes under /playwright/* and optional MCP tools when enabled. Configure browser mode in Settings → General → Playwright (including CDP attachment to an existing browser).

6.4 API recording & replay

Start/stop recording from the UI; recorded HTTP sequences can be exported as Python replay scripts. MCP transport calls (/mcp/*) are generally omitted from replay — prefer direct REST calls in recordings.

6.5 Skills & FastSkills

Build or import skills; when FastSkills MCP is configured, tools such as listing skills and executing tools may be available through that integration.

7. HTTP API & MCP (brief)

• OpenAPI — /openapi.json; interactive Swagger — /docs. Each OpenAPI operationId maps to a candidate MCP tool when MCP is enabled.
• MCP — when enabled: SSE base /mcp, streamable HTTP /mcp-http (see server console on startup).
• Some overlapping endpoints exist (e.g. desktop OCR vs. older text-click snapshot); prefer desktop_ocr_action for unified OCR+click unless you have a legacy integration.

7.1 MCP tool visibility (enable/disable)

Some HTTP routes are hidden from the MCP tool list by default (for example license administration and diagnostics), while remaining available over HTTP or Swagger if enabled. Use Settings → MCP tools to adjust visibility; changes are saved to app data and usually require a server restart (follow on-screen hints).

Technical details (built-in exclusion IDs, JSON schema, non-configurable routes) are documented in the application source repository.

7.2 Monthly quota vs Usage statistics

If you have a subscription license, many API and automation calls count toward a monthly limit. The Usage view uses the same counting rules for graphs, with small exceptions so opening Usage does not skew its own numbers.

Exact path lists and source functions (should_count_request, etc.) are for operators — see the application source repository.

9. Frequently asked questions

Getting started

Do I need MCP or Claude skills to use the product?

No. Core workflows run in the browser UI and over HTTP. MCP and packaged skills are optional when you want an AI assistant to call your local tools.

Do I need Python or Node.js installed?

Not to launch the packaged desktop app. Some optional features (browser automation, FastSkills, skill execution) may require Python or Node.js — see §2.1 and the in-app capability report.

Which browser should I use?

Use a current Chromium-based browser (Chrome or Edge 80+). Legacy browsers may show a blank UI — see §8.6.

Privacy & deployment

Is my data sent to the cloud?

Automation runs on your machine by default. Data leaves your environment only when you configure external AI providers, downloads (models, packages), or integrations you enable.

Is this a hosted SaaS?

No. You run the service locally (or on infrastructure you control). You manage networking, TLS, and who can reach the port.

Features & permissions

Why does mouse or keyboard automation do nothing on macOS?

Enable Accessibility, Input Monitoring, and often Screen Recording for Skill Builder in System Settings, then restart the app. Details: §8.1.

Why is OCR slow the first time?

Vision models download on first use and CPU inference can take minutes on large regions. Subsequent runs are usually faster — see §8.2.

Are VS Code or browser extensions required?

No. Extensions embed the UI but the app must still run locally. See the product site for optional VSIX and browser sidebar downloads.

Still need help?

• Check §8 Troubleshooting and Database MCP build & troubleshooting.
• Prefer in-app hints under Settings and live /docs for your running version.
• Community support is available from the product site; professional support may apply for licensed deployments.

8. Troubleshooting

8.1 Mouse / keyboard does nothing (macOS)

• Open System Settings → Privacy & Security and confirm the running app (Skill Builder, Terminal, or IDE) is allowed under Accessibility, Input Monitoring, and—if you use screenshots or OCR—Screen Recording. Add the app manually if there was no prompt; toggle off/on once if it still fails.
• After changing permissions, quit and restart Skill Builder (and the terminal or IDE if you launched from there).
• Try the remote mouse API with explicit x, y; verify coordinates match primary display resolution.

8.2 OCR slow or “stuck” after log line

• Full-screen Paddle inference can take tens of seconds to minutes on CPU; narrow the region when possible.
• First run downloads models — ensure disk and network are available.

8.3 MCP client cannot connect

• Confirm the server URL and port match your running instance.
• Firewalls / Docker: use the correct host (host.docker.internal vs 127.0.0.1).
• If a service API key is enabled in Settings, only these paths require Authorization: Bearer <secret>: POST /api/run_code, /playwright/*, and /fastskills/*. Other APIs (including /api/host/* and legacy /api/remote/* / /api/action/* aliases) do not use this middleware — do not assume the key gates the whole server.

8.4 CORS or mixed content

• Adjust AUTOMATION_CORS_ORIGINS or browser entry settings if the UI is served from a different origin than the API.

8.5 FastSkills — network errors and offline installation

By default the app starts FastSkills with uvx fastskills, which may download packages from the network (PyPI). If you see TLS errors, timeouts, or “connection reset” during startup or first use, the machine may be offline, behind a strict proxy, or blocked from PyPI.

• Symptoms: Environment check shows FastSkills not running; server log mentions uvx / pip / TLS / handshake failures; skill list or MCP tools hang or fail.
• Fix A — use the same Python as the app: On a build machine with network access, run pip install fastskills into the Python environment that launches this application. Then set AUTOMATION_FASTSKILLS_USE_PYTHON_MODULE=1 (values 1, true, yes, on are accepted). The app will start python -m fastskills with your configured skills and work directories.
• Fix B — custom command (air-gapped): Set AUTOMATION_FASTSKILLS_CMD_JSON to a JSON array of strings, e.g. ["/opt/venv/bin/python","-m","fastskills","--skills-dir","/abs/path/to/skills","--workdir","/abs/path/to/output"]. Use absolute paths if the working directory differs. Invalid JSON is ignored with a log warning; fix the variable and restart.
• Skills layout: Default skills directory is my-fastskills/skills under the application root (or next to a frozen .exe). Ensure that folder exists and contains your skill packages.
• Checklist in the UI: Sidebar → MCP & server utilities → Advanced runs a local capability snapshot (FastSkills process, uvx, env flags, paths).

Monthly quota and the Usage tab both skip diagnostics-style paths (capabilities, machine ID, license endpoints, etc.). See the application source repository for the full list.

8.6 Web UI blank, missing lists, or console syntax errors (browser too old)

The in-app web UI is built with modern JavaScript (for example optional chaining and related language features from around ES2020). Open the app with a current Chromium-based browser — in practice Google Chrome or Microsoft Edge (Chromium) version 80 or newer.

Legacy Windows (for example Windows Server 2012 R2) may only have an old bundled Chrome or Internet Explorer. Typical symptoms: parts of the page never render, sidebar or capability lists stay empty, or the browser devtools console shows parse errors on the first script. This is not fixed by server flags such as --no-paddle; you need a supported browser engine on the client.

Portable Chromium snapshot (example, verified on Windows Server 2012 R2):

We have verified that unpacking the official Chromium continuous-integration zip for Windows x64, revision 1010524, and running chrome.exe from the archive is sufficient to use this UI on Windows Server 2012 R2. Direct download (Google Cloud Storage API; long URL):

https://www.googleapis.com/download/storage/v1/b/chromium-browser-snapshots/o/Win_x64%2F1010524%2Fchrome-win.zip?generation=1654260848325500&alt=media

• This is a snapshot build, not a Chrome stable installer; revision numbers and filenames change over time.
• Snapshots do not auto-update and are not a substitute for vendor-supported browsers for general web use.
• For everyday browsing on older Windows where Google still ships it, an offline Chrome 109 (or enterprise) installer may be preferable; use snapshots mainly for lab / intranet access to this app.
• Verify integrity (checksums, IT policy) before deploying in regulated environments.