Founder & Principal Engineer · Aug 2025 – Present
An AI companion that doesn't forget you
Custom hardware, three memory tiers, and a voice loop tuned to 520ms
Every stage between a child finishing a sentence and hearing a reply, with the time it is allowed to take. The budget totals 800ms and the target 520ms — and the way you hit it is that these stages overlap rather than queue: transcription streams partial text, the model streams tokens, and speech synthesis starts before the sentence is finished.
Recall is split by how fast it has to be and how much it has to hold. All three are queried at once, and a slow tier is dropped rather than waited on — a companion that pauses to remember has already broken the illusion. Emotional memory lives inside the semantic tier as a type, not as a fourth store — the split is by access pattern, not by content.
A circuit breaker moves closed → open → half-open around every external call, and each of L2 and L3 has its own timeout that skips the tier rather than blocking the turn. Degradation is staged, not binary: normal, then memory-degraded, then LLM-degraded, then a canned emergency response — the child is never left waiting on a stalled component to decide what happens next.
Four task modes sit under Normal — math help, homework help, story time, games — and any of them can be entered and exited on its own terms. Emotional support is the exception: it can interrupt every other mode, including itself never being interrupted back, because a child's wellbeing is the one priority the state machine encodes as absolute rather than negotiable.
Eight background agents route through four priority queues, but the pool isn't shared evenly — workers 1 and 2 are reserved for critical and high-priority work only, so safety monitoring always has processing capacity regardless of how backed up personality evolution or index optimisation get. The alternative, one shared pool, would let a burst of low-priority work delay a safety check.
Two requirements in tension
A companion has to remember you. Not "has your name in a system prompt" — actually remember, across months, the way a person does. And it has to answer fast enough that talking to it feels like talking, not like querying.
Memory pushes toward more context. Latency pushes toward less. Almost every design decision here is somewhere on that line.
This one is entirely my own company, so unlike the other two studies I can say whatever I like about it.
Three tiers of memory
L1 in Redis: session state, conversation history, hot facts, and six core memory blocks — persona, human, relationship, session, goals, emotional — that stay in context permanently rather than being retrieved. Read every turn, under 5ms.
L2 in a vector store: episodic, semantic, and emotional memories together, one collection per child, ranked by similarity. Emotional memory isn't a separate tier — it's a memory type inside L2, tagged with which support strategy was used and how well it worked. Under 50ms.
L3 in a knowledge graph: entities, relationships, and bi-temporal facts — what a child believed and when it changed, not just what's true now. Traversed, not scanned, which is what a graph store is good at and a vector store is not. Under 150ms, with a 50ms timeout in the critical path so a slow graph query never holds up a response.
All three are queried in parallel. L1 is required — retry on failure. L2 and L3 race the clock; whatever comes back within budget gets merged in, and the turn proceeds either way.
Decay, then promote
Episodic memories don't live forever by default. They decay on a domain-specific schedule — 30 days for general facts, 90 for anything tagged emotional-support, 180 for educational progress — because a fixed window is honest about what it's for: privacy, not permanence.
What survives is decided by three conditions, all of which have to be true: accessed more than three times, confidence above 0.8, and at least seven days old. Meet all three and a memory gets promoted out of the decaying episodic log and into the permanent knowledge graph as a structured fact. The raw conversation can age out; the fact that survived it doesn't.
That's a deliberate bet, not a compromise. The failure mode I was designing against isn't "forgets everything" — it's the one every long-lived assistant eventually hits, where the important thing from week two quietly falls out of a sliding window and nobody notices until the companion doesn't know you anymore. Decay-and-promote hedges against that without paying to reprocess a growing history forever: the promotion criteria do the triage, so the expensive tier only ever holds what earned its way there.
The honest cost: the thresholds are static and global right now. A fact a child mentions only twice but that matters enormously — a fear, a loss — can still fall through if it's never repeated. Effectiveness-tagged emotional memories get an exemption from decay above a 0.7 effectiveness score, which covers some of that gap. Not all of it.
Reading feeling, not just words
Audio and text are analyzed in parallel on every turn — a wav2vec2 model plus prosodic features (pitch, energy, tempo) on the audio side, RoBERTa plus a keyword layer on the text side — and fused with a 60/40 base weighting toward audio. That split isn't arbitrary: a 2-to-8-year-old's vocabulary is limited enough that tone of voice is the more reliable signal most of the time. But the weights aren't fixed — they scale by each model's own confidence, so a child explicitly saying "I'm scared" can override an uncertain audio read, and a 3-turn temporal-smoothing window keeps one noisy frame from swinging the detected emotion around.
Once an emotion is detected, which of eight support strategies to use — comfort, encouragement, distraction, problem-solving, listening, normalization, positive-memory recall, physical comfort — is chosen by a weighted formula: 30% base fit for the emotion, 50% this child's own historical effectiveness with that strategy, 20% conversational context. The heavy weight on personal history is what makes it personalization rather than a decision tree — after a handful of interactions there's enough signal to know whether a particular kid responds better to being distracted or being heard. A 10% exploration rate keeps the system from locking onto a locally-good strategy too early; without it, the first strategy that works okay would crowd out one that might work better.
Two safety nets, not one
A synchronous keyword blocklist runs before every response and blocks immediately — phrases like "hurt me" or "want to die" can't wait for a model call. An async LLM-based check runs alongside it in a background task, catching what the keyword list misses without adding latency to the conversation, since it never blocks the response.
Neither of those catches a slow-building problem, so a third layer watches across turns and sessions: a background pattern detector flags prolonged high-intensity negative emotion sustained past five minutes, or an emotional trajectory that's been declining across multiple sessions rather than one bad conversation. Critical flags — the keyword-level ones — page a parent immediately. Medium-severity pattern flags roll into a daily digest instead, because not every concerning trend needs a 2am notification.
Lumi can start the conversation
Everything above assumes Lumi is responding. It also initiates — checking in, following up on something from days ago, celebrating a finished story — scored by a six-factor formula: 30% base trigger priority, 20% this child's engagement history with past proactive messages, 15% a recency curve that peaks at 2–6 hours since last contact (too soon reads as clingy, too long reads as having given up), 25% predicted receptiveness for the current hour, 10% topic relevance, and a frequency penalty that grows as the daily cap approaches. Only scores above 0.5 get delivered.
If there's no response, the exit is tiered rather than abrupt — three timeout windows, age-adjusted (toddlers get 8/15/20-second tiers, school-age kids 15/30/45), each with a gentler message than the last, ending in a warm sign-off rather than silence. Frequency starts conservative — three initiations a day, an hour apart minimum — and adjusts automatically from measured per-child receptiveness rather than a fixed setting.
This is the subsystem I get asked about least and would defend most: reactive-only assistants feel like tools. The thing that makes a companion feel like a relationship is that it sometimes reaches out first.
Eight agents, four lanes
"A separate subsystem of background agents evolves the character over time" undersells what's actually eight named agents — memory consolidation, context pre-warming, personality evolution, index optimization, proactive learning, safety monitoring, memory-relevance recognition, conversation-effectiveness analysis — consuming events off Redis Streams consumer groups through a four-worker tokio pool.
The detail worth calling out: two of the four workers are permanently reserved for critical and high-priority work only — safety monitoring and memory consolidation. A burst of low-priority optimization tasks can fill the other two workers completely, and it still can't delay a safety check. That's a small allocation decision with an outsized effect: it makes "critical work never waits behind background work" true by construction instead of by hoping the scheduler behaves.
The 520ms target, 800ms budget
Every stage between a child finishing a sentence and hearing a reply has a budget and a tighter target: 800ms is the ceiling that trips an alarm, 520ms is what the system is actually tuned to hit. Streaming end to end — speech-to-text streaming into the model streaming into text-to-speech — is what makes the tighter number reachable at all; nothing waits for a complete result before the next stage starts. Audio ingestion, transcription, memory retrieval, and reasoning overlap instead of stacking, and a worked-through example under full streaming lands around 480ms to first audio.
That gap matters. Past roughly half a second, people start behaving like they're using a computer: they over-enunciate, they wait, they stop interrupting. Under it, they just talk. The 800ms budget is what keeps a single slow tier from becoming a broken conversation; the 520ms target is what keeps the good conversations feeling like conversation.
Isolation and hardware
The backend is licensed to third-party brands as a white-label platform, which makes per-child memory separation a hard requirement rather than a nice property. There's no relational store to put row-level security on — isolation is structural instead: every child gets their own vector collection and their own graph subgraph, so one brand's devices can't reach another's data by construction, not by convention in application code.
On the hardware side: custom ESP32-S3, low-power, cloud-first — thin capture/stream/play at the edge, with everything that makes Lumi feel intelligent living in the cloud.
What I'd do differently
The promotion thresholds — three accesses, 0.8 confidence, seven days — are global constants, not per-child or per-domain. A memory that would matter enormously to one child but that they only mention once has to clear the same bar as anything else. I'd keep the decay schedule domain-based (it should be) but make the promotion bar adapt to how much a given child actually talks — a chatty kid and a quiet kid shouldn't need the same three mentions to prove something matters.
The proactive system's frequency limits also start from the same defaults for every child before they've said anything. A week of generic caps while the receptiveness model gathers enough signal to specialize is a reasonable cold start, but it's still a week of either being too quiet or slightly annoying before it corrects.