TL;DR

Last article in the series on building Bloom, a multi-agent coaching agent, in one week. Three themes intersect here: memory, smaller models, and validation.

  • Memory: SQLite with structured facts + periodic summaries beats vector search at Bloom’s scale. Extraction runs in the background (fire-and-forget) so it never blocks the response, and compression fires on a threshold of accumulated facts rather than every turn. But the most important finding was different: the read path — injecting recent facts and the latest summary into context — produced more of a sense of continuity than any refinement to writing ever did. Persistent conversation history (short-term) and fact/summary memory (long-term) are distinct things, and conflating them hides the real cause of coherence bugs.
  • Smaller models: switching to smaller local models revealed that model size and instruction-following don’t scale linearly — a seemingly small jump in parameters can be the difference between raw scaffold text leaking into the chat and a clean response. Small models are also more prone to script fixation, even against an explicit user instruction, and “warmth” in tone (emoji, friendliness) is not the same axis as verbosity — one can show up without the other. Practical lesson: whenever you swap models, rerun the evaluation transcripts, because nothing guarantees a correct behavior survives the switch.
  • Validation: before any major rewrite, real transcripts became a fixed evaluation harness with a rubric, scored by an LLM-as-a-judge — turning “feels better” into a measurable regression check. Changes were staged (prompts first, then the tool loop, then memory) to isolate cause and effect, and telemetry moved to per-call instead of per-turn, since loops and delegation made that granularity essential.

The common thread behind all of it: deliberately deciding what belongs to deterministic code versus model judgment — and actively checking that this boundary hasn’t quietly shifted.

Memory, small models, and how to validate any of it

Article 4 of a series on what I learned building a multi-agent coaching agent in one week

I closed the previous article with an open question: when the model behind the agent changes, from a large LLM to a smaller local model, does the balance so carefully built into the prompt survive the switch, or does it silently regress?

This final article in the series pulls together three topics that, at first glance, look unrelated: how Bloom remembers a learner across conversations, what actually changes when the underlying model gets smaller, and how to know — with evidence, not gut feeling — whether an engineering change genuinely improved the system. All three connect through one question, the same question that ran through the whole series: how do you validate decisions in a system that is, by nature, probabilistic?

Memory doesn’t need a vector database to feel like memory

The first instinct, when thinking about “long-term memory” for an agent, is to jump straight to semantic vector search. For Bloom, that would be over-engineering. At the scale of one user per deployment, simple structured records in SQLite, combined with periodic LLM-generated summaries, handle a year’s worth of history without any need for vector infrastructure.

Vector search earns its keep on a genuinely large corpus that requires semantic retrieval. It is not the sensible default for any system that needs memory — it’s a specific tool that has to be justified by the actual scale of the problem, not adopted by reflex.

Fire-and-forget memory extraction

After every conversation turn (except during onboarding), an async background call extracts structured facts from the conversation — preference, barrier, progress, insight — and writes them to a user memory table. This extraction runs outside the latency-critical path: the user never waits on it to get their response.

It’s a simple pattern that separates two responsibilities: responding fast right now, and accumulating knowledge about the person over time. Mixing them means paying, in response latency, for work that never needed to be on the critical path.

Here is how MemoryService implements extractAndStore as an asynchronous background worker in memory.service.ts:

export class MemoryService {
  // ...
  public extractAndStore(userId: string, messages: any[], agentName: string): void {
    // Defined inside a non-awaited async function to prevent blocking the response flow
    const run = async () => {
      const conversation = messages
        .map(m => `${m.role === 'coach' ? 'coach' : 'learner'}: ${m.content}`)
        .join('\n');

      const raw = await llmService.generate(EXTRACTION_PROMPT, conversation, []);
      const cleaned = stripJsonMarkdown(raw);

      let facts: Array<{ fact_type: string; content: string; confidence: number }>;
      try {
        facts = JSON.parse(cleaned);
        if (!Array.isArray(facts)) return;
      } catch {
        return; // silent skip on malformed LLM response
      }

      const validTypes = new Set(['preference', 'barrier', 'progress', 'insight']);
      for (const fact of facts) {
        if (!fact.fact_type || !fact.content || !validTypes.has(fact.fact_type)) continue;
        await LearnerMemoryModel.create({
          id: crypto.randomUUID(),
          user_id: userId,
          fact_type: fact.fact_type as 'preference' | 'barrier' | 'progress' | 'insight',
          content: String(fact.content).slice(0, 500),
          confidence: Math.min(1.0, Math.max(0.0, Number(fact.confidence) || 0.8)),
          source_agent: agentName,
        });
      }
    };

    // Executed immediately and logged if fails, returning control to the client
    run().catch(err => console.error('[Memory] Extraction failed:', err));
  }
}

Threshold-triggered compression, not per-turn

Bloom only summarizes accumulated facts once they hit a threshold (5 raw facts), instead of summarizing every turn or letting facts pile up unchecked. This avoids both bad extremes: summarizing every turn is expensive and redundant; never summarizing produces a pile of raw facts that’s unreadable and expensive to inject back into context later.

Here is how MemoryService.summarize manages the threshold compression gate:

const MIN_FACTS_TO_SUMMARIZE = 5;
const ARCHIVE_AFTER_DAYS = 30;

public async summarize(userId: string): Promise<void> {
  const latestSummary = await MemorySummaryModel.findLatest(userId);
  const since = latestSummary ? new Date(latestSummary.period_end) : new Date(0);

  // Check if we reached the facts threshold
  const facts = await LearnerMemoryModel.findSince(userId, since);
  if (facts.length < MIN_FACTS_TO_SUMMARIZE) return; // Exit early if below threshold

  const factLines = facts.map(f => `[${f.fact_type}] ${f.content}`).join('\n');
  const summaryText = await llmService.generate(SUMMARIZATION_PROMPT, factLines, []);

  // Save the new narrative summary
  await MemorySummaryModel.create({
    id: crypto.randomUUID(),
    user_id: userId,
    period_start: since.toISOString().split('T')[0],
    period_end: new Date().toISOString().split('T')[0],
    summary_text: summaryText.trim(),
    fact_count: facts.length,
  });

  // Archive old processed facts
  await LearnerMemoryModel.archiveOlderThan(userId, ARCHIVE_AFTER_DAYS);
}

The read path is the half that matters most

If I had to choose which to build first, between memory writing and reading, what I learned building Bloom is: reading. Injecting “recent facts + latest summary” into each specialist’s context was, on its own, the change that generated the strongest sense of continuity in the conversation — more than any refinement to extraction or compression.

That’s counterintuitive if you think of memory mainly as a problem of capturing information. In practice, it’s a problem of using that information at the right moment. If you can only build one direction first, build reading before writing.

Here is the context injection implementation on the read path via MemoryService.buildContext:

public async buildContext(userId: string): Promise<string> {
  const [profile, recentFacts, latestSummary] = await Promise.all([
    LearnerProfileModel.findByUserId(userId),
    LearnerMemoryModel.findRecent(userId, RECENT_DAYS, RECENT_LIMIT),
    MemorySummaryModel.findLatest(userId),
  ]);

  if (!profile) return '';

  const details = [
    profile.weekly_time_budget_hours != null ? `${profile.weekly_time_budget_hours}h/week` : null,
    profile.best_time != null ? `best time: ${profile.best_time}` : null,
    profile.confidence_score != null ? `confidence: ${profile.confidence_score}/10` : null,
  ].filter((d): d is string => d != null);

  const goalLine = `**Goal:** ${profile.primary_goal} (${profile.goal_category})`
    + (details.length > 0 ? ` · ${details.join(' · ')}` : '');

  const parts: string[] = [`## Learner Context\n${goalLine}`];

  // Inject recent atomic facts
  if (recentFacts.length > 0) {
    const lines = recentFacts.map(f => `- [${f.fact_type}] ${f.content}`).join('\n');
    parts.push(`**Recent patterns:**\n${lines}`);
  }

  // Inject the latest consolidated narrative summary
  if (latestSummary) {
    parts.push(`**Historical summary (${latestSummary.period_start} – ${latestSummary.period_end}):**\n${latestSummary.summary_text}`);
  }

  return parts.join('\n\n');
}

Two different memories that must not be conflated

Bloom needed two kinds of memory, and treating them as the same thing would have been a mistake:

  1. Persistent conversation history — the last few messages passed on every LLM call, so that “yes” or “option A” resolve correctly from one turn to the next.

  2. Long-term fact and summary memory — a separate store, with a much longer horizon, that survives beyond the current session.

Missing just the first one already breaks basic dialogue coherence — the system literally doesn’t understand what “yes” the user is responding to — regardless of how sophisticated the second one is. These are distinct migrations, and treating them together obscures which one is causing which problem.

Model size and instruction-following aren’t linear

While comparing smaller, local models — with an eye toward getting around rate limits and cost — a data point surfaced that’s worth carrying into any future LLM decision: a 1-billion-parameter model produced visible raw scaffold leakage in the chat (literally text like “Here is my plan for you…” showing up to the user). A model around 2 billion parameters handled the exact same system prompt far more cleanly.

The practical takeaway is that a seemingly small jump in model size can mean a categorical difference in reliability, not a smooth, proportional improvement. You can’t linearly extrapolate “a bit bigger, a bit better.”

A good but slow local model can still break the experience

A heavier local model can deliver equivalent-quality responses and still make the product feel broken. That’s because, in a conversational app like a chat, latency is itself a drag on the experience, independent of how good the response content is. This echoes directly the streaming fix from article 2: multiplying LLM calls (through loops, through delegation) or switching to a slower model has the same perceptible effect on the other side of the screen, and demands the same kind of mitigation.

Smaller models are more prone to script fixation

While testing smaller models like llama3.2 or gemma2, I noticed one of them locked onto scaffolding language from the prompt itself, staying anchored in a career/professional framing even after the user stated, in plain and direct language, a completely different goal unrelated to work.

Practical tip: whenever you switch models, rerun the evaluation transcripts. A behavior that held up perfectly well on one model can silently regress on another, especially anything that depends on the model correctly weighing what the user said against the prompt’s own scaffolding. It’s easy to assume that if a behavior was correct before, it’ll stay correct after a model swap. That assumption is exactly the kind of “decided by code vs. decided by the model” boundary from the first article in this series being crossed silently — only this time by the model swap itself, not by an architecture or prompt change.

Verbosity and formatting are an independent axis from capability

One of the models tested was noticeably “warmer” — more emoji, friendlier tone — but that warmth came packaged with much longer, over-structured responses: nested bullets, entire blocks in bold, closer to a report than a chat message. Taming it took an explicit conciseness instruction.

This finding connects directly to the previous article’s point about soft versus hard limits. There, the same problem (a response turning into a wall of structured text) showed up because of a missing size ceiling in the prompt. Here, it shows up because of a trait intrinsic to the model itself. Two different root causes, the same symptom — which reinforces why “diagnosing the right layer” (the next point) matters so much.

Diagnose the right layer before refactoring

Some interactions were coming across as too rigid, almost “robotic,” with formats that felt off for a chat. As a principle that emerged from the project, the first question I started asking became: which layer is the cause in — architecture, prompt, or data model? In my case, the specific causes were in the prompt, and the problem could be fixed without any architectural change at all.

Stage changes so each layer can be validated in isolation

Bloom’s refactor followed a deliberate sequence: prompts first, then the tool loop, then the memory layer. Each stage was validated in isolation before moving to the next. That stops being “bureaucracy” once you realize it’s what lets you correctly attribute a regression (or an improvement) to the right change. Bundling a prompt rewrite together with an architecture change in the same evaluation round is the perfect recipe for never knowing which one caused what.

Build the evaluation harness before rewriting

Before any major rewrite, Bloom’s real production transcripts — including the unsatisfying ones — became a fixed evaluation set, with a rubric covering naturalness, continuity, adherence to the intended conversational strategy, premature-advice rate, and tone. An LLM-as-a-judge runs that rubric against the “before” version and each “after” version.

That turns “feels better” — a subjective sensation dangerously easy to confuse with confirmation bias — into an actually measurable regression check.

Post-launch review: look for the root pattern, not bug by bug

A structured review after launch turned up four defects that looked completely unrelated: data grounding issues, dishonesty about the real state of calendar sync, an onboarding gate advancing without a genuine answer, and the routing override I described in the first article of this series.

All four were instances of the same pattern: fabricating or skipping information instead of anchoring to what was actually known. Fixing the class of the pattern — not each symptom in isolation — is more durable, and it’s also a reminder that after a launch it’s worth deliberately looking for that kind of common thread instead of just closing tickets one by one.

Per-call telemetry, not just per-turn

Once the number of LLM calls per turn becomes a variable — because of ReAct loops and delegation, as we saw in article 2 — a single “per-turn” cost or latency metric stops making sense without per-call granularity. Bloom started logging latency, status, provider, and token counts per call (including calls inside a tool loop), with endpoints to inspect and reset that telemetry during development. Without an inspectable store, not just logs, that granularity turns into noise instead of signal.

Two levels of testing, neither replaces the other

Finally, unit tests per component catch a broken tool or a prompt-parsing regression; end-to-end tests per full flow catch a routing bug or a state transition that only shows up across multiple conversation turns. An agentic system needs both levels, since neither on its own covers the other’s failure space.

Closing the series

Memory, model choice, and the validation process are, underneath it all, about the same discipline that opened this series: deliberately deciding what’s the responsibility of deterministic code and what’s the responsibility of model judgment — and actively verifying that boundary hasn’t shifted without anyone noticing.

Looking back, almost every ugly bug across building Bloom-for-Learning — the state override from article 1, the prompt leak in the fallback path from article 3, the hallucinated category that onboarding avoided forcing, the small model’s script fixation we saw here — was that same boundary being crossed unintentionally, in different contexts: routing, tone, memory, model choice.

The capstone report closes with a list of future work that I think is worth mentioning as a hook for curiosity, more than as a promise:

  • longitudinal adherence studies comparing Bloom against a simple calendar tracker
  • smaller on-device models (thinking something like Gemma 2B) for full privacy and lower cost
  • expanded MCP support for task managers and document repositories

I don’t know yet if any of these paths becomes the next project. But the real learning from this week of building wasn’t any specific Bloom feature — it was this same measuring stick that ran through all four articles, and that I intend to carry with me into whatever agentic system I build next.