How We Built Handwriting Recognition for 7 Alphabets

We built handwriting recognition that checks how learners write letters and characters in seven scripts. We started with a large language model and dropped it on day one. What we use now is a small sequence network, about 0.8 million parameters per script, that reads the path of your finger. A transparent judge sits on top of it. It runs on our servers without a graphics card and answers in 40–100 ms. On our tests it accepts 95–99% of correct drawings in five scripts and almost never accepts a look-alike character. Here's how we got there, including the mistakes.

Contents

What the Write exercise needs from a machine

In Taalhammer's Write exercise you draw a character with your finger or a stylus, and the app tells you whether it's right. That sounds like a solved problem. Phones have handwriting keyboards. But a keyboard only has to guess which character you meant. We have to judge how you wrote it.

So the machine has to do four things:

1. Recognise the character, in Latin, Greek, Cyrillic, Thai, Arabic, Korean and Chinese.
2. Reject a look-alike. Writing з when the task is э must fail, even if the shapes are close.
3. Reject an incomplete or broken drawing: a missing stroke, an extra line, a mirrored letter.
4. Never fail you for stroke order. It may only give a hint about it.

The last rule comes from research. Writing letters by hand helps you learn them far more than typing or looking: adults learning Arabic letters named 93% correctly after writing practice, against 86% after typing and 83% after looking (Wiley & Rapp, 2021). Freehand writing works better than tracing over a model (James & Engelhardt, 2012). The evidence that one particular stroke order helps is mixed. A fixed order hurt very young children (Merritt et al., 2020), and stroke-order animations lowered recognition in adult learners of Chinese (Hou & Jiang, 2022). So the exercise uses an empty grid instead of a model to trace, and the checker judges the shape, not the order.

On top of that come the engineering constraints. The same drawing must always get the same verdict. A check must cost nothing and take well under a quarter of a second. And it all has to run inside our ordinary API servers.

Attempt 1: ask a large language model

The quickest prototype sent each drawing as an image to a large language model. It asked which character it was and how well it was written. It worked, in a way. It recognised a clean a with 0.97 confidence and gave a sloppy one a low score.

It also took 2.7–4.4 seconds per drawing and cost money on every press. Worst of all, the same drawing got a different score each time. A learner who presses "check" twice and gets two verdicts stops trusting the exercise. We dropped it the same day.

Attempt 2: match templates, then let a judge decide

Our second version was a template matcher. It used the $P point-cloud recogniser (Vatavu, Anthony & Wobbrock, 2012), which turns a drawing into a cloud of points and finds the closest stored template. We already used it in our alphabet courses.

The first real test was a hand-traced Thai ก. It scored zero. Our automated tests had been too kind. The template, built from a font, was broken into four fragments. Jittered copies of a broken template still match each other perfectly, so the tests passed while real drawings failed.

We rebuilt the pipeline around measurements rather than one distance. For each drawing we compute about a dozen numbers: how close it is to the target versus the nearest other character, how much ink is missing or extra, whether there's a stray stroke, and how many dots and strokes it has. A logistic regression, one per script, turns those numbers into a verdict.

Two lessons came out of this stage:

• Thresholds hit a ceiling. Tightening a single distance threshold cut defects only by rejecting good drawings. Correct Latin letters accepted fell from 81% to 68%.
• Stroke order must stay out of the verdict. When we let synthetic writers vary their stroke order, acceptance of correct Korean and Chinese drawings collapsed to 40% and 47%. Our stroke measure still depended on order. An order-free version, which pairs up strokes whatever their sequence, brought both back to about 71%.

Adding a tiny image network

The cheapest big win was a small convolutional neural network (CNN). It looks at a 32×32 picture of the drawing and outputs how likely each character is. It has about 190,000 parameters, weighs about 270 kB per script and runs in plain numpy, so the API needs no machine-learning library. Its confidence became two extra inputs for the judge.

Look-alike acceptance dropped from 1–3% to almost zero. Correct Thai drawings accepted rose from 83% to 98%, Korean from 75% to 89%.

From pictures to the pen's path

A picture throws information away. On a phone we know where each stroke starts, which way it goes and where the finger lifts. Recognising from that path is called online handwriting recognition, as opposed to offline recognition from an image.

Our first sequence network read 96 points along the path through four convolutional layers. It weighed at most 316 kB and ran in about a millisecond. We compared it with the image network on the same data, script by script. The sequence network won for five scripts, and the image network kept Cyrillic and Thai for the time being.

Chinese was the problem. With 1,200 characters, many differing by a single stroke, the engine accepted only 63% of correct drawings on our tests. It ranked the right character first only 46% of the time.

The audit: our own numbers lied

Before trying a bigger model, we audited how we measured. We found two errors that made us look better than we were:

• The validation split leaked. We had split drawings, not writers, so the same person's handwriting sat on both sides of the test.
• "Random" real writers weren't random. They were the first few writers in a file.

Fixing both made the numbers worse before anything got better. The audit also showed where the real loss was. The judge, not the network, was the bottleneck. It had been fitted on synthetic drawings, so it read "right but unsure" as "wrong". It rejected about 40% of the drawings the network had recognised correctly.

The sequence network that worked

The second sequence network was the turning point. It's what runs today.

The input. We resample the drawing to 160 points. Each point carries its position, the pen's movement, its orientation (in a form that doesn't change when a stroke is drawn backwards), the jump the pen makes in the air between strokes, and a pen-up flag. The old network had thrown the pen-up jumps away.

The network. It's a residual one-dimensional CNN, 128 channels wide. Its layers are dilated, spaced 1, 2, 4 and 8 points apart, twice over. So each output sees the whole path. The old network saw only 17 of its 96 points at once. Instead of a plain classifier it learns two prototypes per character and scores a drawing by how close it is to each.

Training. We perturb stroke order the way real people do: swapping neighbouring strokes now and then, and rarely reversing one. Random distortions are strong at first and fade to zero by the end of training, so the network finishes on clean shapes. A Chinese model trains for 40 epochs.

Shipping. For inference we fold the normalisation layers into the convolutions and export the weights to numpy. A model is 1.5–2.1 MB per script, about 0.8 million parameters, and a forward pass takes about 5 ms on a CPU.

Two changes around the network mattered as much as the network itself:

• The network proposes the candidates. The template matcher now only re-ranks and explains. Korean recognition jumped from 53.4% to 95.5% with this change alone.
• The judge may not reward mess. Left free, the logistic regression learned that real writers leave lots of missing and extra ink. It gave those features positive weights and rejected 42% of perfect textbook drawings. We now constrain the sign of every weight. Missing ink can only lower a score.

For Chinese, correct drawings accepted went from 63.3% to 96.7%, and recognition from 46.1% to 97.7%. But the network had become so good at reading messy writing that it read incomplete drawings too. Defects accepted rose from 15.6% to 29.5%.

Teaching it to say "that's not a character"

The fix was an extra reject class: the network learns what a broken drawing looks like and can answer "none of these". Getting it right took five versions.

大 minus one stroke is 人. The first reject network started rejecting a correct 人. We had labelled "大 with a stroke missing" as a defect, and that drawing is 人.

So we filter the defects. An older network without a reject class reads every generated defect, and if it sees another real character with at least 50% confidence, the defect is dropped. About 16% of them were.

Learners write like the textbook. Native adults don't. The next version misread 6% of perfect textbook characters, reading 爸 as 职 with 99.9% confidence. A network trained on fluent, cursive adult writing has rarely seen the careful textbook form, which is exactly what learners produce. We added 200 synthetic drawings per character, 40% of them near-textbook, and then 12 exact textbook copies per character.

The final generator makes six kinds of defect, in fixed proportions: a missing stroke (28%), an extra stroke (23%), a stroke borrowed from another character (14%), a truncated drawing (14%), a mirrored one (7%) and a rotated one (7%). Scribbles came later.

The Chinese model that ships accepts 97.0% of correct drawings, lets through 9.2% of defects and recognises 97.6% of characters, with 100% of exact textbook drawings accepted. On 14 October we moved every other script to the same recipe.

Where the training data comes from

We name only data that anyone can download:

Source	What it contains	Licence
UNIPEN	real pen recordings of Latin letters	CC BY 4.0
UJI Pen Characters	real pen recordings of Latin letters	CC BY 4.0
Omniglot	drawings from 20 people per alphabet, including Greek and Cyrillic	MIT
Make Me a Hanzi	stroke-by-stroke data for 9,000+ Chinese characters	Arphic PL / LGPL
Noto and other open fonts	letter shapes in every script	SIL OFL

For Thai, Arabic, Korean, Greek and Cyrillic there's no free collection of real pen strokes big enough to train on. For Thai there's no public online collection at all. So we built a synthetic hand. We aligned the model shapes onto 3,285 real UNIPEN drawings and measured how people distort letters: a typical tilt of ±13°, slant, stretched proportions and a wobble of about 3% of the letter size. About 8% of writers join strokes that the model lifts. Real writers turned out two to three times messier than our first guess. The assumption behind it: the same wrist writes Thai and Latin. So one fitted hand writes every character in every script, thousands of times.

Fonts turned out to be poor teachers. They describe outlines, not movements. When we built Korean stroke data from font outlines, 36% of syllables had the wrong number of strokes. We rebuilt the stroke data from teaching standards.

Training on one laptop

Everything trains on one laptop's graphics chip (an M1 Pro with 16 GB of memory shared with the rest of the machine), overnight. It's 19 times faster than the laptop's CPU for this network: 8 minutes per million drawings instead of 151. A Chinese run takes two to seven hours. Latin takes under half an hour.

The laptop is also a working computer, so training runs under a watchdog. It watches memory pressure and pauses or checkpoints the run before the machine starts swapping. Every training and test run is recorded with its settings and results. Over five days we logged more than 150 of them.

Some of what went wrong:

• A data-building step hung for seven hours. Worker processes had been forked after the graphics library started, and they deadlocked. The only alarm we'd set was "tell me when it succeeds". Now every wait also has a time limit.
• Too many maths threads made data generation take 23 minutes instead of one. Each worker now gets one thread.
• Our comparison tool scored a new model on the old model's measurements. Three verdicts were wrong and had to be re-run.

How we measure, and how a model wins

Every script has a labelled test corpus in two parts that never share a writer:

• TUNE is the only part we make decisions on.
• HOLDOUT is only for reporting. If we tuned on it, it would stop telling the truth.

Each part holds correct drawings from neat to sloppy, look-alikes (a good drawing of another character), defects and scribbles. We measure:

Measure	What it means	Target
Good accepted	correct drawings, neat to fairly sloppy, that pass	≥ 95%
Look-alike accepted	another character accepted as the target	≤ 0.5%
Defect accepted	an incomplete or broken drawing that passes	≤ 10%
Recognition	the right character is the first guess	≥ 85% (Chinese ≥ 97%)
Textbook	an exact textbook drawing passes	≥ 99%
Speed	95% of checks answered within	250 ms

These measures fold into one score on TUNE. It rewards correct drawings accepted, adds a smaller reward for sloppy ones and subtracts steep penalties above 2% look-alikes and 10% defects. A candidate replaces the live model only if it beats that score by more than 0.01 and stays at or under 0.5% look-alikes on HOLDOUT. Otherwise the old model, its calibration and its settings are restored byte for byte.

This rule cost us nicer-looking numbers more than once. A larger network for Latin, Greek and Cyrillic (192 channels, 60 epochs) cut HOLDOUT defects to 8–9%, but lost on TUNE. We kept the smaller one. A bigger network wasn't the limit.

The results for each script

Results on HOLDOUT:

Script	Characters	Good accepted	Look-alike accepted	Defect accepted	Recognition
Thai	74	99.0%	0.1%	2.0%	99.3%
Arabic	100 forms	97.7%	0%	2.6%	97.8%
Chinese	1,200	97.0%	0%	9.2%	97.6%
Cyrillic	66	95.8%	0%	10.4%	98.2%
Korean	924 syllables	95.4%	0.1%	2.1%	95.5%
Greek	48	93.2%	0.4%	13.2%	94.1%
Latin	62	91.9%	0%	11.6%	90.4%

For Thai, Arabic, Korean, Greek and Cyrillic, the test drawings come from our synthetic hand and a handful of hand-made drawings. Read those rows as "how well the checker handles the hands we can simulate".

Speed was a threading problem, not a model problem. On a busy machine, numpy's default maths library started a full pool of threads for every small matrix product, and they fought over the cores. With one thread per process, the median check fell from 280–450 ms to 42 ms for Latin, 60 ms for Greek and 101 ms for Chinese. In the production image, the 95th percentile for Latin went from 76 ms to 26 ms.

What it can't do yet

• Thai has never met real Thai hands. Its numbers are all synthetic. The image network it replaced was trained on scanned Thai handwriting, but the sequence network wins on our tests and is the one that runs. Learners' drawings will tell us more.
• Latin, Greek and Cyrillic are below target. The hardest case is a short extra stroke across a letter, which no measurement separates from a fluent join. Our TUNE sets are small, so single cases decide comparisons. A larger TUNE corpus comes next.
• Chinese still accepts some drawings with a missing stroke.
• We have no real learner drawings yet. With consent, they'll become the test set that matters most.
• Japanese isn't supported yet. We're working on it. Kana are the natural next step, and kanji share most of their strokes with Chinese.

For the learner's side of one of these scripts, read our guide on how to write Arabic.

For more on how we check that a measuring tool tells the truth, read which TTS voices get pronunciation right. For scoring speech instead of handwriting, see how pronunciation assessment works.

FAQ

Does the handwriting recognition run on the phone?

No. Your phone sends the strokes, and our servers check them in plain numpy, without a graphics card. A check takes 40–100 ms, so it feels instant.

Why not use ChatGPT or another AI model?

We tried. A large language model took up to 4.4 seconds per drawing, cost money on every check and gave the same drawing different scores. A small specialised network is faster, free per check and always gives the same answer.

Does it check stroke order?

It sees the order and direction of every stroke and uses them for hints. It never fails a drawing for its order, because the research doesn't show that one particular order helps you learn, and when order counted, correct drawings by other hands were rejected far too often.

What data was it trained on?

Real pen recordings of Latin letters (UNIPEN, UJI Pen Characters), drawings from Omniglot, stroke data from Make Me a Hanzi and open fonts. For most scripts the training drawings come from a synthetic hand fitted to real writing.

How accurate is it?

On our HOLDOUT tests it accepts 95–99% of correct drawings in Thai, Arabic, Chinese, Cyrillic and Korean, and 92–93% in Latin and Greek. It almost never accepts a look-alike (at most 0.4%).