Learning from evidence.
A system for reasoning about latent capabilities from observable evidence.
How do we know what we know?
A capability cannot always be observed directly. We see evidence of it through actions, decisions, responses and performance.
When an engineer diagnoses a complex system failure, or when a strategist navigates conflicting priorities, their intrinsic cognitive capability is not directly readable by an external sensor.
Instead, the external world receives discrete empirical evidence: a successful decision, an error under time pressure, a sequence of trade-offs. From these observable signals, an inference system must construct and maintain a formal belief about what remains hidden.
Figure 00.1 · Discrete evidence flows into continuous posterior belief state.
What should the system believe?
This establishes the core research problem before introducing mathematical formalisms: how should discrete evidence transform into continuous belief?
INTERACTIVE DEMONSTRATION
Interactive Inference: Sequential Updating in Real Time
INFERRED ESTIMATE
Starting at neutral uninformative midpoint (0.500).
"The problem is not simply estimating a capability. The problem is knowing how much evidence supports that estimate."
Can a system learn a hidden capability from evidence?
How our research developed through three deliberate stages — each motivated by the limitations discovered in the previous one.
We initiated this inquiry with a straightforward empirical question:
"If we observe whether a participant succeeds or fails at a task, how should a system update its estimate of their underlying capability?"
Rather than jumping to an overly complex model, the research progressed through three distinct phases. Crucially, each phase was motivated by an empirical limitation discovered in the previous one:
Exponential Moving Average
A sequential scalar estimator that adjusts a single point estimate with each new observation.
Bayesian Capability Inference
Represents capability as an explicit probability distribution carrying both estimate and variance.
Adaptive Task Selection
Leverages the current belief state to actively choose the next observation that maximizes information gain.
Starting with the Baseline: The Exponential Moving Average
Our exploration began with sequential scalar estimation—updating beliefs step-by-step as each observation arrives.
Before exploring probabilistic architectures, we evaluated the fundamental sequential tracker: an Exponential Moving Average (EMA). In this baseline model, the system maintains a single scalar estimate and updates it in real time upon observing each binary task outcome.
The update rule is lightweight and computationally direct: starting from a neutral midpoint , each observed binary task outcome produces an instantaneous error residual , nudging the estimate by learning rate :
INTERACTIVE EQUATION
Testing the Learning Rate Parameter (η)
EMA gives us an estimate. It does not give us uncertainty.
Consider two participants who both register an estimate of . Participant A achieved this after 3 observations. Participant B achieved this after 80 observations. The scalar EMA represents both as identical, discarding sample size and concealing how little we know about Participant A.
The Shift: What if capability wasn't a point?
From a fragile scalar point estimate to an explicit probability density curve.
To address this limitation, we changed the fundamental representation of human capability:
THE CONCEPTUAL MORPH
Scalar point estimate · Zero variance representation
What if capability was a belief?
A distribution preserves all plausible capability values, weighting them by empirical support.
Bayesian Capability Inference & Belief State
Modeling continuous belief through conjugate Beta distributions.
In the Beta-Binomial framework, latent capability is governed by a Beta distribution:
Evidence supporting success
Increments by +1 for each successful task performance.
Evidence supporting failure
Increments by +1 for each failed task attempt.
We begin with the neutral prior , which represents total uniform uncertainty:
EVIDENCE CONSOLE
Posterior State: Beta(1, 1)
Positive (α)
1
Negative (β)
1
Estimate E[θ]
0.5000
Uncertainty (σ)
± 0.2887
Experiment 01: Consistent Positive Evidence
Replaying our initial benchmark experiment with 11 consecutive successes.
In our first empirical trial, we subjected the inference engine to uninterrupted positive evidence: consecutive task successes.
Starting from uninformative prior , every success increments parameter , shifting the posterior distribution steadily rightward while concentrating density:
EXPERIMENT 01 · CONSISTENT EVIDENCE
Beta(1, 1) → Beta(1, 1)
Observations (n)
0
Posterior State
Beta(1, 1)
Estimate E[θ]
0.5000
Uncertainty (σ)
± 0.2887
Experiment 02: Mixed & Contradictory Evidence
What happens when the evidence disagrees? Testing the model against real-world contradictory signals.
Real human performance is rarely flawless. We presented the model with a contradictory 9-observation sequence (6 successes, 3 failures):
EXPERIMENT 02 · MIXED (PS-002B)
Beta(1, 1) → Beta(1, 1)
Observations (n)
0
Posterior State
Beta(1, 1)
Estimate E[θ]
0.5000
Uncertainty (σ)
± 0.2887
Compare the Experiments: What Changed?
Overlaying Consistent vs Mixed posterior distributions on a shared coordinate canvas.
SHARED COORDINATE SYSTEM
Posterior Comparison Overlay
Evidence changes belief
Posterior estimates respond directly to accumulated observations.
Direction of evidence matters
Consistent observations produce substantially tighter estimates than contradictory observations.
Uncertainty is observable
Bayesian inference yields an explicit variance metric (σ) that is absent in scalar baselines.
More evidence increases certainty
Variance shrinks as total sample volume accumulates.
Same evidence. Different representations.
The centerpiece comparison: What does the Bayesian model tell us that the EMA doesn't?
EXPERIMENTAL CENTERPIECE
Synchronized Dual-Model Runner
Point Estimate
0.500
No uncertainty metric
Estimate ± Uncertainty
0.500± 0.289
THE RESEARCH INSIGHT
What does the Bayesian model tell us that the EMA doesn't?
Uncertainty.
We do not claim that Bayesian inference is universally superior. Rather, Bayesian inference provides an explicit representation of uncertainty that the scalar EMA baseline fundamentally cannot provide.
Research Limitations: What this model does not know
A credible research programme makes its boundaries and assumptions explicit.
The current models are intentionally minimal baseline formulations. They operate under specific simplifying assumptions that will be expanded in subsequent research phases:
Binary Observations Only
Responses are strictly coded as y ∈ {0, 1}, ignoring partial credit or nuances.
No Task Difficulty Calibration
Currently treats solving an easy task as equivalent to solving a hard task.
Static Capability Assumption
Assumes latent θ remains constant during the assessment without fatigue or learning effects.
Synthetic & Unit-Test Scope
Evaluated on synthetic benchmark experiments, not yet calibrated on large-scale human cohort data.
What should we observe next? The Adaptive Loop
From passive observation to active information acquisition.
Once a system possesses both an estimate and an explicit measure of uncertainty, the fundamental research question changes:
"Suppose we currently believe θ ≈ 0.64. Which task should we administer next to learn the most about this person?"
Figure 15.1 · Active inference loop: Belief informs next measurement · Measurement informs belief.
The goal is not simply to administer more questions.
It is to select the observation expected to reduce uncertainty most.
Interactive Adaptive Engine
Manipulate current belief to see how the system reasons about optimal task selection.
ACTIVE INFERENCE SIMULATOR
Optimal Task: Task C (Medium)
CANDIDATE TASKS ACROSS DIFFICULTY SPECTRUM
Lab Notebook & Unit Test Suite
Exposing development logs, test assertions, and reproducibility checks.
DEVELOPMENT EVIDENCE
Experiment Log PS-002B & Pytest Verification
Boundary updates, step responses, learning rate scaling.
Conjugate updates, variance decay, Beta(7,4) reproduction.
100% test pass rate across core inference engine.