May 17, 2026, 11:40 PM

brain/raw/articles/5_menthal models.md

@@ -0,0 +1,224 @@
+---
+title: "The 5 Core Mental Models for AI Agents: Harness & Memory (Deep Dive + Action Plan) - BPMS Team"
+source: 
+author:
+published:
+created: 2026-05-14
+description:
+tags:
+  - "clippings"
+---
+
+## Overview
+
+This page distills the five consensus mental models expert teams use to take agents from demos to production, then maps them to a concrete fraud-analysis use case built with Gemini CLI, MCP tools, and BigQuery. Use it as a reference for design reviews, implementation planning, and capability audits.
+
+Table of Contents
+
+## Mental Model 1 — The Model Is the CPU. The Harness Is the OS.
+
+> "Strip away the harness and you have a raw language model guessing its way through your codebase. Add the right harness and you have a system that ships production code."
+
+## Core Idea
+
+LLMs are powerful but general-purpose reasoning engines. Quality emerges from the harness (the “OS”) that shapes inputs/outputs, tools, policies, and runtime controls. Changing the harness often changes outcomes more than swapping models.
+
+## Analogy Mapping
+
+- **Model → CPU:** raw intelligence (e.g., Gemini 2.5 Pro)
+- **Context Window → RAM:** working memory (MCP context, GEMINI.md)
+- **Harness → OS:** tools, routers, policies, safety gates
+- **Agent → Application:** your fraud pipeline (prompts + tools + flow)
+- **Memory → Disk/SSD:** persistent state across sessions
+
+## Why It Matters
+
+- Different harnesses around the same model produce large score deltas on realistic tasks.
+- Slimmer, better-curated toolboxes often outperform maximalist sets due to lower decision entropy.
+
+## Design Implications
+
+- Make the “OS layer” explicit: state store, checkpoints, retries, idempotency, and traceability.
+- Bias for simple, tight tool catalogs with strong schemas and pre/post-conditions.
+- Invest early in observability of agent steps, not just final replies (see Observability references below).
+
+Your Gemini CLI fraud agent already has a proto-harness (MCP tools, GEMINI.md, SQL generation). The missing “OS” capabilities are: state persistence, checkpoint/resume, and run-level tracing.
+
+## Mental Model 2 — Context Is the Product
+
+> "Every harness artifact answers the same question: What does the agent need to know before it writes a single line of code?"
+
+## The Context Stack
+
+5\. Procedural Memory ("How we do things here")
+
+4\. Semantic Memory ("What is true right now")
+
+3\. Episodic Memory ("What happened before")
+
+2\. Session State ("Earlier this session")
+
+1\. Working Context ("Right now: last N turns + active tools")
+
+## The Core Tension
+
+Trade off in-context stuffing vs. out-of-context retrieval:
+
+- Full context: zero latency to recall, but costly and “lost-in-the-middle” risks.
+- Sliding window: cheap, but causes session amnesia.
+- RAG (vector): good semantic recall; must tune chunking, reranking, and filters.
+- Hybrid vector + graph: best for entities/relations; requires ontology upkeep.
+
+## Design Implications
+
+- Separate “always-in” system context from “retrieve-when-relevant” memory.
+- Layer vector search (semantic) with graph traversal (relational correctness).
+- Use LLM-managed memory policies: what to store, how to compress, when to forget.
+
+For the query “877-417-4551,” persist: (a) semantic fact “shared phone links to 2,097 entities,” (b) episodic trace “investigation run with outcomes,” (c) procedural heuristic “for shared-phone queries use 3-hop traversal.” Your existing ArangoDB graph is a natural semantic + episodic backend.
+
+## Mental Model 3 — Separate the Doer from the Judge
+
+> Agents are unreliable self-graders. External verification (computational tests, an evaluator model, or humans) is essential.
+
+## Verification Spectrum
+
+- Level 1: Self-check (cheap; catches formatting/syntax; misses subtle issues)
+- Level 2: Computational checks (schema/type/linters/tests; low cost, medium power)
+- Level 3: Inferential LLM judge (semantic/design review; medium cost)
+- Level 4: Adversarial evaluator agent (high cost; highest defect catch rate)
+- Level 5: Human-in-the-loop (HITL) (doesn’t scale; gold standard)
+
+## Design Implications
+
+- Insert feedforward constraints (contracts, schemas, tool preconditions) and feedback sensors (tests, monitors, audits) in the harness.
+- Never allow the authoring agent to “approve” high-impact actions; split duties.
+
+Add for your BigQuery SQL agent: (1) pre-exec schema validation, (2) guardrails on result size/runtime; auto-ask for confirmation if estimated rows > threshold, (3) HITL gate for destructive or high-impact ops. Aligns to D&B oversight requirements and internal harness patterns observed in production agents.
+
+## Mental Model 4 — Memory Is Three Cognitive Tiers, Not One Bucket
+
+## Three Tiers
+
+- **Episodic:** past events/traces/outcomes; chronological, bitemporal annotations; ideal in temporal graphs or run logs.
+- **Semantic:** facts/entities/relationships; hybrid structured rows + embeddings; conflict detection and merging are key.
+- **Procedural:** instructions, playbooks, heuristics; static files (AGENT.md) or runtime prompt updates with guardrails.
+
+## Design Implications
+
+- Pick backends by tier, not “one DB for all.” Graph stores fit semantic and episodic; files or prompt-update APIs fit procedural.
+- Implement conflict resolution for semantic memory (compare new facts to graph entries, merge/update/flag).
+- Track “memory provenance” so the harness can justify recalls and updates.
+
+Mapping for your fraud agent: “2,097 entities linked to 877‑417‑4551 in Florida” → Semantic (ArangoDB). “Last investigation on 23 kwi 2026 found shell factory” → Episodic. “Shared-phone queries → 3-hop traversal first” → Procedural (GEMINI.md or runtime system-prompt rule).
+
+## Mental Model 5 — Graduated Autonomy: Earn Trust Through Verification
+
+## Gradient of Autonomy
+
+- Level 0 Read-only
+- Level 1 Suggest-only
+- Level 2 Approve-then-act
+- Level 3 Act-then-review (audit)
+- Level 4 Autonomous (bounded)
+- Level 5 Fully autonomous
+
+## Design Implications
+
+- Start lower, promote autonomy based on proven reliability (evals + memory of track record).
+- Constrain the environment: tight tools, strict contracts, and yardsticks for promotion/demotion.
+
+D&B’s internal governance aligns to 6 autonomy tiers. Use episodic memory for track record (“47 runs, zero false positives”) to graduate; use semantic memory for sensitivity flags (“touches PII”) to require HITL; use procedural memory to encode newly learned safe behaviors.
+
+---
+
+## Applying the 5 Models to Your Gemini CLI + BigQuery Fraud Agent
+
+## Target Architecture Additions (“OS Layer”)
+
+- **State & Checkpointing:** Persist per-run state and decisions (e.g., Postgres/Arango + run\_id) so long tasks can resume after failure. See internal examples using LangGraph state + Postgres checkpointing in production agents.
+- **Observability:** Trace prompts, tool calls, inputs/outputs, token/cost, and branch decisions. Capture row estimates for SQL, retrieval stats for memory calls, and validation outcomes. Align to AI observability guidance: infra + model/agent + quality/safety telemetry with correlation.
+- **Guardrails:** Schema-aware SQL builder; preflight “EXPLAIN” or dry-run checks; result-size and latency thresholds → evaluator/HITL gates; redaction and policy filters for sensitive fields.
+
+## Memory Plan (Three Tiers)
+
+- **Episodic:** Investigation traces with inputs, traversals (e.g., phone → entities → addresses), evidence URIs, outcomes, and evaluator judgments. Store bitemporally to enable regression and “what changed since last run?” analyses.
+- **Semantic:** Entity facts in ArangoDB: phones, persons, businesses, addresses, edges with weights and recency. Add conflict detection rules when new facts disagree with old; record source and confidence.
+- **Procedural:** GEMINI.md rules + runtime “policy injects” (e.g., “On shared identifiers: 3-hop traversal; for row estimates > 1M, ask confirmation; for PII tables, require HITL”). Maintain a changelog of procedural updates with who/what justified the change.
+
+## Harness Hardening (Doer vs. Judge)
+
+- **Feedforward:** Strict tool schemas, SQL builder with typed columns and table contracts; entity resolver with disambiguation prompts; scope checks before any heavy query.
+- **Feedback:** Computational checks (lint/validation), evaluator LLM for semantic soundness on traces, targeted HITL for high-impact actions or low-confidence decisions.
+
+## Graduated Autonomy Rollout
+
+- **Phase A (L1):** Suggest-only SQL + rationale; mandatory evaluator review for all queries.
+- **Phase B (L2):** Approve-then-act for read queries under thresholds; HITL for anything exceeding table sensitivity or row/latency caps.
+- **Phase C (L3):** Act-then-review for non-sensitive reads with strong historical precision; auto-rollback heuristic if evaluator flags anomalies.
+
+## Context Strategy
+
+- Working context: last N tool calls + current hypothesis + active graph nodes.
+- Session state: rolling compressed summary of this investigation run.
+- Episodic/semantic recall: hybrid retrieval—vector similarity on notes/explanations plus graph traversal for facts; re-rank by provenance and freshness.
+
+---
+
+## Implementation Blueprint
+
+## Milestone 1 — OS Layer Foundations
+
+- Add run\_id and checkpoint tables; store step-level inputs/outputs and decisions.
+- Integrate AI tracing (prompts, tool calls, latencies, cost) and expose searchable logs.
+- Harden SQL tool: typed schema, table allowlist, safe templates, EXPLAIN pre-checks.
+
+## Milestone 2 — Memory (ArangoDB-first)
+
+- Design graph schema for phone↔entity↔address relations with edge attrs: weight, recency, source, confidence.
+- Implement semantic write path with conflict detection/merge rules; provenance is mandatory.
+- Add episodic store for run traces linked to graph nodes/edges; enable “delta since last run.”
+
+## Milestone 3 — Doer/Judge Split
+
+- Introduce computational validators (schema, row-estimate caps, sensitive-table guards).
+- Add evaluator LLM for semantic quality on investigations (fact sufficiency, alternative explanations, leakage risks).
+- Wire HITL gates for flagged conditions; log reviewer decisions back into episodic memory.
+
+## Milestone 4 — Graduated Autonomy
+
+- Define promotion criteria (precision/recall on eval sets, incident-free runs, operator feedback).
+- Automate demotion on evaluator regressions or anomaly spikes.
+- Track autonomy tier in agent state; show in UI with audit trail.
+
+---
+
+## Risk Controls and Operational Readiness
+
+- **Data governance:** Align table access to sensitivity tiers; mask/redact PII in context; retain audit logs.
+- **Cost control:** Token usage and query-cost meters; context compression; hybrid retrieval to avoid over-stuffing prompts.
+- **Reliability:** Timeouts, retries with backoff; idempotent tool calls; partial-progress resumes via checkpoints.
+- **Evaluation:** Seed test sets and continuous evals for accuracy, grounding, and safety; track scorecards per model/tool revision.
+
+## Quick-Start Artifacts
+
+- **AGENT.md (procedural seed):** investigation steps, guardrails, escalation criteria.
+- **Tool contracts:** JSON Schemas for SQL generation, graph traversal, entity resolution.
+- **Evaluator prompts:** structured critique rubric (soundness, sufficiency, safety, performance impact) with pass/block decision.
+- **Runbook views:** dashboard panels for per-run trace, thresholds triggered, autonomy tier, and memory writes.
+
+## FAQ
+
+No. Start with a single agent and strong harness (tools, validators, evaluator). Add a separate evaluator agent later if quality gaps remain or scale requires parallel critics.
+
+Use both. Vector for semantic recall of notes/past narratives; graph for factual correctness and relationship traversal. Your ArangoDB gives you a strong graph spine; add vectors for unstructured artifacts with provenance links.
+
+Define quantitative gates (precision/recall on evals, zero critical incidents over last N runs, latency/cost SLO adherence) plus qualitative operator feedback. Store outcomes in episodic memory as evidence.
+
+## Follow-ups
+
+- Create a 1-page “Mental Models Notebook” PDF with diagrams for team onboarding.
+- Deliver a gap analysis specifically for your Gemini CLI + BigQuery fraud agent with a 30–60–90 day plan.
+- Deep-dive design for “ArangoDB as memory backend” (schema, indices, traversal templates, conflict-resolution rules, and retrieval orchestration).
+
+## References