How Taalhammer builds — and tests — its adaptive language engine
This is a deep-dive for people who care about the engineering behind language learning, not a product tour. If you want to know what actually happens between "you subscribe to a language" and "you see the next sentence", and how we check that the sentence was worth showing you, read on.
We have been rebuilding the core of Taalhammer around a single rule, and then spending most of our effort proving that the rule is being followed. This article covers both halves: how the engine is built, and how we test whether it teaches.
The one rule
The design target is Stephen Krashen's comprehensible input, also written i+1 (Krashen, Principles and Practice in Second Language Acquisition): you acquire a language when the input is one small step beyond your current level. It is understandable because almost everything in it is already known, plus exactly one new element.
We turn that into a hard constraint on every sentence the engine produces:
Each sentence introduces exactly one new element. If the new element is a grammar structure, every word and every other structure in the sentence must already be known. If the new element is a word, every structure and every other word must already be known.
The interesting part is that we do not trust the language model to obey this. We check it.
What the engine tracks: features, not lessons
Traditional courses ship a fixed list of lessons in a fixed order. Our engine has no lessons. It tracks features, and each feature is an independently schedulable unit of knowledge:
- Words, frequency-ranked and CEFR-selected. The beginner list leads with the nouns and verbs a learner actually needs, not a raw frequency dump.
- Grammar structures, a CEFR-graded inventory of 200+ per language, weighted A1 to C2, each with example sentences.
- Phrases and chunks, multi-word units taught as one element (
avere fame, not "to have hunger"), because fluent language is built from collocations, not single words. - Verb forms, stored as
lemma@tense(for examplesein@present), so "knowing the conditional of essere" is a different fact from "knowing its subjunctive", each tracked on its own. - Numbers, dates and times, a small generative system drilled separately from the word list, because "half past seven" and "nineteen sixty-eight" follow rules, not frequency.
State is split the way the rest of the system splits it. Your personal learning state lives in Postgres (accounts, scheduling, per-feature confidence), and the content itself (sentences, collections, audio references) lives in Elasticsearch.
Finding your level without asking
You never tell us your level. The engine discovers it with Bayesian adaptive testing on two axes at once, grammar level and vocabulary frontier, the same statistical machinery as a computer-adaptive exam (item response theory over a two-dimensional ability estimate). Each probe updates a posterior over both axes, and the next probe is chosen to reduce that posterior's uncertainty fastest. In simulation this converges to within about ±0.4 CEFR in roughly 11 probes, and it recovers if you were having a good or a bad day at the start.
Generating a sentence you can actually understand
Once your level is known, producing the next sentence is a small pipeline:
- Pick the target: the one new element to introduce (a structure, a word, a phrase, a verb form, or a number), plus a palette of words you already know to build the rest of the sentence from.
- Generate: a large language model (a GPT-class model) writes a sentence that exercises the target using only the known palette.
- Prove it: a per-language feature extractor, built on spaCy dependency and morphology analysis plus hand-written rules, re-reads the generated sentence and lists every structure and every word lemma it contains. If anything unknown leaks in beyond the single target, the sentence is rejected and regenerated.
This is the step that turns "soft i+1" into a checked one, and it is worth being precise about what the check does and does not prove. It is one-directional: it verifies that nothing unknown leaked in, not that the target itself was detected. Roughly a quarter of grammar structures are not detectable by the extractor at all, and its coverage of the structure inventory runs from about 39% to 76% depending on the language, so "no leak" means "no leak of the detectable subset". It is a real constraint that catches real violations. It is not a proof. 4. Gate the grammar: a second model, deliberately different from the generator (a model checking its own output is blind to its own mistakes), certifies that the sentence is grammatical and that the translation is faithful, repairing or rejecting it otherwise.
About 70% of sentences pass the extractor on the first try, at an average of two attempts. The rest are regenerated, and the roughly 30% that still cannot be built cleanly are served with a flag rather than silently. Those flagged sentences are genuine i+1 violations the verifier caught, either the model adding a structure above the frontier or an extractor false positive. We serve them flagged because an honest flag beats a silent leak, but they are served, and they count against us.
There is a length cap too. The point of an i+1 sentence is that one element is under test, so the target gets the complexity budget and everything around it stays short and plain. A hard grammar structure does not get buried inside a stack of adjectives, and a new word does not get lost in incidental syntax.
Serving: N, then N+1
At runtime the engine runs in two modes, and it flips between them on its own:
- N, repetition. It serves what spaced repetition says is due: words and phrases shown as-is (flashcards), and structures re-tested inside fresh sentences. A single review sentence can combine several due items built entirely from known material, so one sentence drains several reviews at once.
- N+1, introduction. When nothing is due, it introduces exactly one new element, a real i+1 sentence.
Day one is all N+1, because there is nothing to review yet. After that, each session mixes the two. The learner is never asked "review or learn?". The engine decides from what is due.
Spaced repetition and the memory model
Scheduling is where the "when to show it again" decision lives. The engine uses FSRS-6, the current version of the Free Spaced Repetition Scheduler, the same open-source, machine-learned memory model that Anki now ships as its scheduler. FSRS represents each item with three quantities (the DSR model): retrievability (your probability of recalling it right now), stability (the number of days for that probability to fall from 100% to 90%), and difficulty (1–10). The forgetting curve is a power function, R(t) = (1 + FACTOR · t/S)^decay, because real forgetting has a long tail that a plain exponential misses.
What makes it a machine-learned model rather than a formula is how its 21 parameters are set. They are trained the way a small recurrent network is trained: back-propagation through time with maximum-likelihood estimation over a learner's review log, with a card's stability carried forward across its reviews like a hidden state. The parameters stay interpretable (they are retrievability, stability and difficulty), which is what separates FSRS from a black-box network and from the fixed 1980s formulas it replaces. The public FSRS-6 weights were fitted on roughly 700 million reviews from about 20,000 volunteer users; we run our own py-fsrs 6.3.1 with the 21-weight parameter set. To schedule, we pick a target recall of 0.9, and FSRS solves its forgetting curve for the moment recall would drop to that level; that becomes the next interval. A review does the most work when it lands just before you would have forgotten, the desirable-difficulty effect.
On top of the scheduler sits a per-feature confidence value between 0 and 1, revised on every observation with an exponential moving average. A directly tested correct answer moves it a lot; a word reused as scaffolding moves it a little. A "correct but slow" or "almost" answer lowers confidence, a contradiction signal, and when confidence drops below threshold the feature is marked lapsed: it leaves the set of things we build sentences from, and it becomes a re-teaching target.
That last part had a bug worth mentioning, because our own testing caught it. A failed item is rescheduled to come back almost immediately, but the same failure also flips its state to lapsed, and the review query was only serving items in the known state. So difficult items were being scheduled for re-teaching and then never actually served. We found it because a test we had just written flagged that only 19% of failed items were resurfacing. The fix: the review queue now serves any due item, shakiest first, so the things you got wrong are exactly the things that come back.
How we test whether any of this teaches
Here is the part we care about most, because it is the part most language-learning systems skip.
Layer one, rubrics. We keep an executable specification: dozens of acceptance criteria split into hard invariants (a single violation is a bug, for example a learner never sees the same generated sentence twice) and softer targets (for example, the share of items that are phrases should track the level's Core mix). Every change is run against them.
Layer two, clean-flight simulations. We drive six simulated learners, one per CEFR level from A1 to C2, through the live service from an empty database, ten days of about fifty answers each, and score every served item against every rubric. When a metric fails, an attribution pass sorts the failing items into causes (a real leak the generator produced, versus a false positive from the extractor, versus a measurement artifact), so a red number becomes a ranked worklist instead of a mystery.
Layer three, the outside learner. Every rubric is a proxy. Passing them does not prove a human learns. So we built an external test that uses none of the engine's internal signals.
The outside learner, in detail
The idea: train a learner from a random initialisation on nothing but the served stream, and measure whether it acquires the language. "From scratch" is the load-bearing constraint. We deliberately do not use a pretrained large language model. A network like GPT-2 or a German BERT has already read billions of German tokens, so a few thousand more shift its predictions by noise; it would report its prior, not our curriculum. This is the premise of the academic BabyLM challenge: train from scratch on a tiny corpus (here about 2,300 sentences), no pretrained weights allowed. A blank slate that learns German efficiently from the served stream is evidence about the curriculum, independent of anything the engine believes about itself.
The learner is a per-feature memory model, not a neural network. Every feature it meets (a word, a grammar structure, a verb form, a numeral, or a phrase) carries one state variable, a memory half-life h, and its probability of recalling that feature falls with elapsed time t as p = 2^(−t / h). This is Half-Life Regression, the model Duolingo published for exactly this problem (Settles & Meeder, A Trainable Spaced Repetition Model for Language Learning, ACL 2016). Two update rules drive it, one for each half of N/N+1:
- Comprehensible input (the N+1 gate). The learner absorbs a sentence only if at most one or two of its features (a new word, or a new grammar structure) are still unknown. That is the operational form of Krashen's i+1, and of the finding that a reader needs to know about 95–98% of the words in a text to comprehend it (Hu & Nation, "Unknown vocabulary density and reading comprehension", Reading in a Foreign Language 13(1), 2000). A sentence past that budget is wasted exposure, the way a C2 sentence is wasted on a beginner.
- Forgetting and spacing (the N half). An absorbed feature's half-life decays with time and lengthens on review, and it lengthens most when it is reviewed at moderate recall, the desirable-difficulty effect that FSRS is also built around. A feature seen once and never revisited is forgotten; one reviewed at the right moments sticks. A model that never forgets could not show any of this, so the spaced-repetition half of the algorithm would be invisible to it.
Training is online: one sentence at a time, in served order, so the learning curve can be read directly. The features the learner tracks come from the same extractor the engine uses to prove i+1, so it is scored on exactly what the engine claims to teach. As a check that the finding is a property of the curriculum and not of one model, we ran the same streams through an independent from-scratch model, a character-level n-gram language model (Witten-Bell smoothing, which handles German morphology with no out-of-vocabulary problem); under the same comprehensibility gate it points the same way.
Then we run an A/B with the content held fixed and only the order changed: the exact same set of sentences fed three ways, the adaptive N/N+1 order our engine served, a fixed CEFR ladder ("Core order"), and random (that same set of sentences shuffled into no particular order, not random or nonsense text). Same daily load, same held-out test, same learner. Only the sequencing and review timing differ.
The result
Measured on a full A1 to C2 trajectory (one simulated learner climbing the whole ladder, about 2,300 unique sentences), across 12 resampled runs, after a two-week forgetting gap. The learner here acquires every feature the engine teaches: words, grammar structures, numerals, verb forms and phrases, each as its own memory trace.
| Curriculum | Features still recalled after 14 days | Exposure wasted (incomprehensible) |
|---|---|---|
| Adaptive N/N+1 (our engine) | 174.1 ± 3.2 | 23.8% |
| Fixed CEFR ladder | 170.6 ± 4.8 | 25.3% |
| Random | 94.4 ± 12.0 | 59.6% |
The dominant result is comprehensible input. Both the adaptive engine and a fixed CEFR ladder leave the learner recalling about 80% more than random and wasting less than half the exposure. That gap is large and statistically clear (adaptive over random: +79.7 ± 10.3 features retained, and 35.8 ± 4.2 percentage points less wasted). Once grammar structures count toward whether a sentence is understandable, a stream that is not ordered for comprehensibility becomes incomprehensible almost 60% of the time, and the learner takes little from it.
Between the two curricula that are ordered for comprehensibility, the difference is small. Adaptive retains +3.5 ± 5.0 features more than the fixed ladder, which is not statistically significant on this test. A well-built CEFR course is itself a comprehensible-input ordering, so on pure sequencing of a fixed set of sentences the adaptive engine and a good ladder come out close to even.
That first result holds the content fixed and varies only the order, which isolates sequencing. But the adaptive engine's distinctive move is not sequencing a fixed deck; it is generating different content for each learner, one step beyond what that specific person already knows. So we ran a second experiment that varies the content: the engine's per-learner stream against the real Taalhammer Core 1–4 course (the fixed sequence a learner would otherwise follow, about 6,900 items from A1 to B2), both served through the same spaced-repetition scheduler, so the only difference is the content itself.
The design choice that makes this work is that we do not test a single learner. We simulate a range of learner types, because the honest answer to "is adaptive better?" turns out to depend on who is asking. New features each learner still recalls after two weeks:
| Learner | Adaptive | Core | Winner |
|---|---|---|---|
| A1 (absolute beginner) | 20 | 34 | Core |
| A2 | 21 | 14 | Adaptive |
| B1 | 15 | 12 | Adaptive |
| B2 | 9 | 7 | Adaptive |
| C1 (past the B2 ceiling) | 12 | 7 | Adaptive |
| C2 (past the B2 ceiling) | 14 | 7 | Adaptive |
| B1, spiky (strong grammar, weak vocab) | 24 | 15 | Adaptive |
| B2, spiky | 21 | 13 | Adaptive |
(Early figures from single simulation runs; the magnitudes are being firmed up with repeated runs and confidence intervals, so read the pattern, not the decimals.)
It is a clean crossover, and a more honest story than "adaptive always wins." A purpose-built beginner course beats the adaptive engine for a true A1 beginner, because a course built for beginners is hard to beat for a beginner. From A2 upward the engine is ahead, and by more the further a learner sits from that average beginner: modest on the course's own levels, large where a one-size course structurally cannot fit the person. A learner past the course's B2 ceiling has no new material left in a course that stops at B2; a lopsided spiky learner needs vocabulary that a grammar-ordered course cannot single out. The one-line version: for the average learner on the course's own path a well-built course is a strong baseline, and adaptivity is worth what it reaches that a fixed course cannot.
What we are not claiming
- The personalisation figures are early: single simulation runs per learner type, not yet confidence-bounded. Read the pattern (who wins, and roughly by how much), not the exact numbers, until repeated runs put error bars on them.
- The sequencing confidence intervals cover variation in the test split and the model's parameters, but come from a single generated stream. The final figures need several independently generated streams.
- The forgetting curve is a reasonable default, not yet calibrated to Taalhammer's real review history, so the comparison between curricula holds, but the absolute retention numbers are model-relative.
- It is a model, not a person. The last word belongs to a human retention study.
What is next
The direction we are pushing hardest is the space of learners itself: mapping which types of learner gain most from adaptivity. The result above already shows why this is the right question. "Is adaptive better?" has no single answer; it depends on who the learner is, and the honest output of this research is not one headline number but a map of who benefits and by how much. We have run the absolute beginner, the on-path learner at each level, the beyond-ceiling learner and the spiky learner; next are off-pace learners (faster or slower than a fixed course assumes) and finer profiles in between, each with repeated runs so the margins carry confidence intervals rather than single-run point estimates. We would rather report that map than an average that hides it.
If you build language-learning systems and want to compare notes on any of this, the feature extractor, the verify-and-regenerate loop, or the outside-learner methodology, we would genuinely like to hear from you.