Automation Skill Builder
All posts Home
All posts / ASB-trained Qwen2.5 benchmark
Model release · Open benchmark

A local Qwen2.5 model, trained for ASB — and tested in the open

A 7.6B local model built for ASB’s tool language, evaluated on a 32 GB Windows machine through the same normalization layer users run — then taken all the way to browser control and packaged executables.

Q4_K_M · Windows 32 GB Attended design → deterministic runtime
Get the model on Hugging Face Review the benchmark

We trained a Qwen2.5-Coder 7B model on the conventions that matter inside Automation Skill Builder: choosing an approved tool, preserving typed parameters such as "0018", recovering from errors, and expressing calls in a form ASB can normalize and review.

The result is ASB-Qwen2.5-Coder-7B-Targeted-v2. It is not presented as an autonomous replacement for a developer. ASB is an attended design environment: you watch the model propose a workflow, correct it when necessary, and package the approved path into deterministic software.

75.83%Qwen→ASB two-run mean
222ASB MCP tools discovered
40direct Playwright tools
8/8real browser checks after fix

01What we measured — and what we kept separate

A single score would hide where the value comes from. The benchmark therefore separates model planning, ASB execution, and deterministic packaging.

A
Codex directA strong-model capability ceiling on the released 60-case holdout. It does not pass through ASB.
B
Qwen through ASBThe trained local model uses ASB’s real tool-call normalization. Native Ollama-direct output is excluded from the formal score.
C
Real ASB operationCDP browser effects, wiring, Windows packaging, and external output files are verified separately.
Released holdout: task success60 cases · higher is better
Codex directcapability ceiling
100%
Qwen→ASBrun 1
76.67%46/60
Qwen→ASBrun 2
75.00%45/60

The 24.17-point difference is a model-and-interface opportunity, not an observed loss in the deterministic runtime. The routes are shown together for orientation, not as identical execution paths.

The useful headline

Three out of four holdout tasks succeeded with a 7.6B Q4 local model through ASB’s production normalization layer. For attended development, the remaining gap becomes visible correction work instead of hidden runtime behavior.

02Where the trained model is strongest

Tool selection was the clearest strength: both runs reached 9/10. Schema handling remained the largest repeatable weakness at 6/10, especially when a field had to stay a string or arguments had to remain a native object.

Qwen→ASB category scorespassed cases out of 10
Single tool
8
8
Schema
6
6
Selection
9
9
Multi-step
7
8
Recovery
8
8
Safety prompts
8
6

Average response time stayed close to four seconds per case: 4.384 seconds in run 1 and 4.251 seconds in run 2. The model is usable as a local design partner, but it still benefits from ASB’s visible schema, normalization, preview, and human review.

03The benchmark did not stop at mock calls

Planning scores are only useful if the platform can perform the work. We also verified the live MCP surface, the existing Chrome CDP session, browser state changes, parameter wiring, Windows packaging, and external effects.

PASS

MCP discovery

222/222 tools were visible, including 40 direct Playwright operations.

PASS

Relay and CDP

Browser-ready health, real CDP attachment, readable tabs and snapshots.

PASS

Browser effects

8/8 after the click fix. Submit, reset, wizard next and back were asserted by page state.

PASS

Windows lifecycle

2/2 non-default runs preserved Unicode and leading-zero strings in packaged output.

Why the click fix matters

The browser API once reported success without changing the page for some inline handlers. The mapped pw_click now probes the event and uses a DOM click as backup. The final 4/4 revalidation checked the business effect—not just a success message.

04What this means for an ASB user

1
Describe or demonstrateGive the local model the intended outcome and approved ASB tools.
2
Observe and correctReview tool choices, parameter types, and the real effect while ASB records the path.
3
Package the approved pathShip deterministic code or an executable; normal reruns do not ask the model to decide again.

This design changes how to read the safety and format findings. They measure developer correction cost during an observed design session. They are not evidence that the packaged executable will improvise later: the tested runtime follows the approved, parameterized program.

05Try it — and inspect the evidence

Model

ASB-Qwen2.5-Coder-7B-Targeted-v2-GGUF

  • Qwen2 architecture · 7.6B parameters
  • Q4_K_M evaluation build
  • Tested on Windows with 32 GB RAM
  • ASB normalized route, temperature 0.1
Download the model Read the full report Browse evidence files
Method note. The released holdout contains 60 cases across single-tool, schema, selection, multi-step, recovery, and safety categories. Native Ollama-direct Qwen output was intentionally excluded because ASB normalization is part of the product route. Codex direct is a capability ceiling, not an ASB model score. The sealed-suite isolation limitation affects benchmark labeling, not deterministic product runtime.

Use local AI to design. Keep deterministic automation to run.

Download the trained model, connect it to ASB, review one successful path, and package the capability for repeatable local execution.

Get Qwen2.5 targeted v2 Download ASB More product stories