Skip to main content

AI Assistant

Model Benchmarks

Model Benchmarks is a scorecard that shows how well ThreatWeaver's AI features are performing for each kind of task — so you can see, in numbers, whether the AI is actually earning its keep before you rely on it for something important.

What it's for​

  • See performance, not just output — pass rate, response time, and cost for each task type, instead of just trusting a generated result.
  • Compare across task types — recon, exploitation analysis, remediation, and triage each get their own table, so you can judge each on its own terms.
  • Watch it stay current — the scorecard refreshes on its own, so the numbers you're looking at are recent without doing anything.

Where to find it​

Model Benchmarks lives under AI Labs in the sidebar, alongside the other AI tools.

Who can see this

Model Benchmarks requires a specific permission. Admins and security analysts can see it by default; other roles — a read-only viewer, for example — may not. If the page doesn't appear for you, ask an administrator to grant access.

Reading the scorecard​

The page shows one table per task type — the kinds of work ThreatWeaver's AI handles, such as Recon, Exploit, Remediation, and Triage. The four core task types always appear first, in that order; any additional task types your organization has enabled appear afterward.

Model Benchmarks — one table per task type, each row showing pass rate, latency, cost, and sample size

The columns, explained​

ColumnWhat it shows
Provider and ModelWhich AI provider and model ThreatWeaver routed that task type to.
Pass RateThe share of runs that met the quality bar for that task type, as a percentage. Rows within a table are ranked with the best pass rate at the top.
Avg LatencyHow long a typical run took to come back — in milliseconds, or seconds once it gets slower.
Avg CostThe average cost of a single run for that task type.
Sample SizeHow many runs the numbers are based on. A larger sample means a more reliable average.
RecordedThe date the benchmark figures were captured.

A table may also show a small version label next to its title. That's just a tag for the current round of scoring — useful mainly if you're comparing numbers you saved earlier against what you see today.

If a task type has nothing yet​

A task type with no benchmark runs yet shows a short note instead of a table. If the whole page has nothing to show, it tells you no benchmarks have been recorded — check back once your AI features have been in use for a while.

Auto-refresh​

The scorecard keeps itself current on its own:

  • It automatically re-checks for new numbers about once a minute.
  • It also refreshes whenever you switch back to the tab after being away.
  • A Last updated timestamp under the page title shows exactly when the numbers you're looking at were last pulled.

You don't need to reload the page to see new results — just leave it open, or come back to it later.

Common workflows​

Workflow: checking model performance for a task type​

  1. Open AI Labs → Model Benchmarks from the sidebar.

    Step 1 — the Model Benchmarks page under AI Labs

  2. Find the table for the task type you care about — for example Remediation.

    Step 2 — the Remediation task type table

  3. Compare the rows: the top row has the best pass rate. Check its Avg Latency and Avg Cost to see what that performance costs you in time and spend, and Sample Size to gauge how much data the numbers rest on.

    Step 3 — comparing pass rate, latency, and cost across rows

  4. Note the Last updated timestamp so you know how fresh the numbers are — the page will keep refreshing on its own while you have it open.