How to Evaluate Multi-Model AI Decisions

This document is the methodological anchor for Smart Logic AI’s multi-model governance research program (SL-RG-METHOD-2026.1). Publication date: 2026-09-10. Last updated: 2026-09-10.

It defines prompt-set construction, model freezing, disagreement classification, materiality criteria, completeness rules, and reproducibility requirements for dependent reports. It is not a scored dataset and contains no fabricated benchmark scores.

Dependent studies: agreement/divergence, disagreement taxonomy, traceability, and override study. Authority context: multi-model governance, consensus vs. divergence, model vs. decision governance, and framework vs. platform.

Methodological conclusions only — no scores.

• Quantitative benchmark results across the program remain forthcoming; this document publishes method, not measured rates.
• Reproducible multi-model evaluation requires frozen prompts, frozen model versions, and explicit disagreement labels before any rate is publishable.
• Materiality criteria separate soft divergence from decision-critical conflict so escalation policy can be tested later without inventing frequencies.
• NIST AI RMF separates Govern, Map, Measure, and Manage functions — measurement without governance context is incomplete^[1][2].
• OECD AI Principles and ISO/IEC 42001 motivate accountable, documented AI management practice; OMB M-25-21 reinforces federal governance expectations for agency AI use^[3][4][5].

Program reports share a common evaluation spine: (1) define the decision context and risk class; (2) freeze the prompt inventory; (3) freeze eligible models and versions; (4) capture raw outputs and human actions; (5) label disagreement and refusal classes; (6) score ledger completeness; (7) only then compute rates.

Steps 5–7 produce quantitative outputs only after a measured corpus is released. Until then, dependent pages must state measurement not yet available or methodology published; benchmark results forthcoming.

This page remains the citation anchor for method ID SL-RG-METHOD-2026.1.

Dataset size: Methodology document — not a scored dataset.

Methodology document ID: SL-RG-METHOD-2026.1.

This report does not release a scored corpus. Dependent benchmarks inherit pending status until measured data is published under this method.

Prompt sets: versioned inventories with inclusion/exclusion rules, prohibited-request classes for refusal studies, and change control when prompts are amended.

Model freezing: record provider, family, identifier, version, and decoding configuration needed for rerun. Unfrozen comparisons are non-publishable for rates.

Disagreement classification: apply the taxonomy in the disagreement dataset report; adjudicate decision-critical and policy classes with dual review.

Materiality: map conflict classes to required human authority before analyzing “agreement rates” as governance outcomes.

Traceability: apply the field checklist in the traceability benchmark before including an event in any rate denominator.

Human actions: use the action taxonomy in the override study for accept/modify/override/reject/escalate coding.

Refusal coding: use the protocol in the refusal benchmark so non-responses do not silently bias agreement scores.

A quantitative result may be published on smartlogicusa.com only when all gates below are met. Until then, pages must not invent rates.

1. Register the study under SL-RG-METHOD-2026.1.
2. Freeze prompts and models; store identifiers.
3. Capture outputs and human actions with Decision Ledger fields.
4. Label disagreement, refusal, and action classes.
5. Apply completeness gates.
6. Compute and review rates internally.
7. Publish only after evidence audit allows quantitative release.

Use this methodology as the internal standard for any public Smart Logic AI research claim about multi-model agreement, refusal, override, traceability, or cycle time.

Separate framework adoption (NIST AI RMF, ISO/IEC 42001) from platform execution — see framework vs. platform — and separate model inventory from decision authorization — see model vs. decision governance.

Do not present pending measurements as completed benchmarks in client or federal materials.

This document does not contain scores, sample sizes, or measured rates. Absence of numbers is intentional.

Organizational policies differ; materiality thresholds must be localized before interpreting future rates.

External references establish governance context; they do not validate unpublished Smart Logic AI metrics^{[1][2][3][4][5]}.

Smart Logic AI publishes methodology before metrics. SmartSolo provides the operational architecture — multi-model comparison, human authorization, and Decision Ledger evidence — against which future measured corpora can be evaluated.

Program reports link back here as the methodological anchor (SL-RG-METHOD-2026.1). Quantitative releases will update dependent pages only after evidence audit clearance.

Continue with multi-model AI governance and consensus vs. divergence for operational guidance while benchmarks remain forthcoming.