Benchmark framework update: baselines, ablations, MQM, WCAG ACT, and LLM TCO

Context#

• Date: 2026-01-28; category: benchmark.
• Recent work reflects knowledge organization into classification shards, expansion of self-recognition/evaluation capabilities, and new governance-oriented orchestration patterns.
• Today’s workspace differences are configuration-only; no new benchmark runs landed. This post codifies the benchmarking framework using repository knowledge artifacts.

Baselines first#

Every machine learning project benefits from a baseline. Options include:

• An existing non-ML or rule-based solution
• Simple statistical heuristics (e.g., averages)
• A prior production system

A strong baseline anchors progress and prevents regression.

Data discipline for evaluation#

• Maintain strict splits: training, development (validation), and evaluation (test).
• Never tune on the evaluation set; reserve it for final measurement.
• Track data lineage to avoid leakage across splits.

Ablation studies: isolate and measure#

Definition: Systematically remove or vary specific components, modules, layers, or features to quantify their contribution. Core principles:

• Isolate variables: alter one component at a time; hold all else constant.
• Use consistent training and evaluation conditions.
• Report with confidence intervals when possible.

Practical workflow:

• Identify components to test
• Design controlled variations
• Run and compare against the baseline

Example applications include keeping perception/classification stable while varying a localization module, or toggling auxiliary losses/features in a controlled manner.

Bilingual QA and MT: MQM-based quality assessment#

• Integrate MQM to evaluate machine translation output quality within bilingual QA workflows.
• Use MQM to surface specific error types and guide post-editing or model iteration.
• Combine MQM with task outcomes to connect language quality to downstream utility.

Accessibility benchmarking via WCAG and ACT Rules#

How to measure consistently:

• Apply ACT Rules to operationalize WCAG checks for reliable, comparable results.
• Interpretation:
• Failure of an ACT Rule implies the corresponding WCAG success criteria are not satisfied.
• Passing indicates no failures were detected by that rule.

WCAG 2.2 highlights (Level AA to implement):

• 2.4.11 Focus Not Obscured (Minimum)
• 2.4.13 Focus Appearance (Minimum)

Benefits of aiming for 2.2:

• Enhanced accessibility and mobile usability
• Legal risk mitigation (e.g., Section 508 now incorporates WCAG 2.2; exceeding ADA Title II’s typical 2.1 target can reduce exposure)

Empirical guidance to prioritize fixes:

• Six issues account for ~96% of detected errors
• Low-contrast text appears on ~80% of pages
• Increased ARIA usage correlates with more detected errors; use ARIA carefully
• Sector patterns: some public-sector, education, social media, technology, and personal finance sites tend to perform better; sports, shopping/e-commerce, and style/fashion tend to be worse
• Overall trend: detectable WCAG failures decreased only slightly year over year (e.g., ~95.9% to ~94.8%)

LLM inference TCO: what to measure#

To compare serving configurations, specify:

• Model and accelerator: model size (e.g., 70B), GPU type (e.g., H100/H200), and VRAM per device
• Performance: prefill TPS per device, decode TPS per device, latency SLO
• Cost envelope: instance pricing, utilization targets, and scaling strategy

Tie TCO to SLOs so cost comparisons reflect production-relevant constraints.

Governance for benchmarked models#

Implement model governance across stages:

• Responsible candidate assessment (feasibility, ethics, business alignment)
• Model inventory and metadata management
• Controls across development, validation, deployment, and monitoring
• Evidence collection to support audits and change management

Putting it together: a practical plan#

• Establish baselines for each task (non-ML and prior-system baselines at minimum)
• Define clean splits and lock the evaluation set
• Add MQM scoring where bilingual QA or MT is in scope
• Adopt ACT Rules for WCAG conformance checks; prioritize the six most prevalent issues first
• Run ablations to attribute gains to specific components
• Track LLM inference TCO with explicit prefill/decode throughput and latency SLOs
• Record decisions and results in the governance process for reproducibility and auditability

Status today#

• No benchmark code changes detected in today’s workspace; configuration updates only.
• Next cycle will focus on executing the above measurement plan and publishing baseline and ablation results alongside TCO and accessibility scorecards.