We pointed our document parser at a public dataset of a few thousand PDFs and got this back for one of them:
Digitalmarketingprofessionalexperiencedinsales,marketingandaccountmanagement.
Every line of every page looked like that. Nothing crashed, no check failed, and both of our parsers dutifully tried to make sense of one enormous word per line.
A PDF does not promise spaces between words. It stores characters and where to draw them; some generators write a space character and some do not, and when they do not, every space you see in extracted text is a guess made from the gap between two characters. If you have to guess, measure each gap against the spacing of its own line, keep your library's original rule as a floor so the change can only ever add spaces, and check the result against an answer key instead of eyeballing it.
Why PDF spaces aren't always there to extract
A PDF page is a drawing program. Its text operators say "use this font, at this size, draw these characters starting here". Many generators also draw an actual space character between words, and when they do, extraction is easy: the space is in the stream and the extractor copies it.
Nothing requires that space character, though. A generator can just as well draw "Digital", move the pen a little to the right, and draw "marketing". On screen the result looks identical. In the text stream there is no space at all, only a horizontal gap.
So every PDF text extractor has a rule for turning gaps into spaces, and each library picks its own number and its own unit. Ours is the Ruby gem pdf-reader (version 2.15.0), and its rule is hardcoded and simple: if the gap between two characters is at least 0.2 times the font size, insert a space. Every number in this article is relative to that rule, not a universal constant. (Gaps are easiest to reason about in multiples of the font size, called "em" units, because they then hold for any font size.)
What went wrong
The dataset's PDFs all came from one HTML-to-PDF exporter (wkhtmltopdf 0.12.4). In every export we inspected, the exporter drew no space characters at all. It placed every word by position, and in our case it left about 0.14 em between words, under the 0.2 em the library needed. Inside a word the measured gaps were even slightly negative, about 0.07 em of overlap, partly from kerning and partly from fonts that state a glyph wider than they draw it.
With every word gap below the threshold, the library joined every word on a line. When we compared extracted words with the true text, the median document scored 9%, because almost no "word" we produced existed in the original.
Two things made this nasty:
- It failed silently. The text wasn't empty, so nothing that checked for a failed extraction fired. Both parsers received a page of plausible-looking characters and did their best.
- None of our test documents had it. Not one of the 15 real-world documents we had been testing with showed the problem. It only surfaced because we tried a dataset made by a different generator.
The obvious fix, and why it was wrong
The obvious move is to lower the threshold. At 0.1 em nearly all of the dataset's words came apart (98% captured; more on the rest below). We also re-extracted 15 real-world documents with the new threshold, expecting no change at all, and two of them changed:
- A name typeset with wide letter-spacing came out as single letters, "J A N E D O E". Its letters were deliberately spread apart, by less than the old 0.2 rule's threshold but more than the new 0.1.
- A date range, "2022–2026", gained spaces around its dash. The dash's own side spacing read as a gap. That sounds cosmetic, but our parser treats a spaced dash as a separator between a title and an organisation, and an unspaced one as a range. One extra space changed what the line meant.
Any single number has this problem. Letter-spaced headings need a high threshold; position-placed words need a low one. Both kinds of document exist, so no constant can serve them both.
A gap is wide relative to its own line
What tells a word break apart is not how wide the gap is, but how much wider it is than the gaps between letters on the same line. In the letter-spaced name, every letter sits the same distance from the next, and the break between first and last name is wider still. In the exported documents, letters overlap and words sit 0.14 em apart, a jump of about 0.2 em over the typical letter gap.
So the rule we shipped looks at each line on its own:
- Take the median gap between neighbouring characters on the line. That is the line's normal letter spacing, whatever font, kerning or tracking produced it.
- A gap is a word break if it is at least 0.08 em and at least 0.08 em wider than that median.
- Keep the library's original rule as a floor: a gap of 0.2 em or more is always a space, exactly as before.
- Never add a space (below the floor) next to a dash, before closing punctuation or after an opening bracket. Their side spacing reads as a gap, and a space in those places is usually wrong. One exception: when the gap after a dash already clears the floor, the gap before it is judged by rule 2 like any other. These exclusions are conservative choices from the documents we had, not typographic law, and the exception was a second look forced by data (below).
In illustrative code (simplified: the real version also skips gaps where the PDF already drew a space, and knows all six dash characters):
def is_word_break(gap, font_size, line_median_gap, left_char, right_char, gap_after_right):
if gap >= 0.2 * font_size: # the library's rule, unchanged
return True
spaced_dash = right_char in "-–—" and gap_after_right >= 0.2 * font_size
if not spaced_dash and (left_char in "-–—([{" or right_char in "-–—)]},.;:!?%"):
return False # side bearings, not words
return gap >= 0.08 * font_size and gap - line_median_gap >= 0.08 * font_size
The floor matters more than it looks. Because every space the old rule produced is still produced, the change is additive for spaces: it can only add a space where the old extraction glued two words, never remove one. That rules out one whole class of regression, a lost space, but not the other: a space added where none belongs could still split an identifier, a URL or a letter-spaced word that extracted correctly before. So "is this safe to ship?" went from a question about every PDF in existence to a narrower one we could actually answer: look at every space the new rule adds, and nothing else.
The exclusion that was only half right
The spaced_dash line in the code above was not in the first version. A
reviewer asked what happens to "Senior Engineer — Platform Team" when both
gaps around the dash sit below the floor, and the honest answer was that the
dash rule glues it. We measured instead of arguing, and the data was worse
than the hypothetical. In this exporter the gap after a visibly spaced dash
clears the 0.2 em floor, so the library spaces it, while the gap before the
dash sits below the floor, where our rule refused. Result: 739 lines in the
sample came out as "expert- Pivot", and none came out right.
The fix follows from what the floor already tells us. When the gap after a dash clears the library's own threshold, the library has already decided the dash is spaced on that side, so the gap before it can be judged by the normal relative rule instead of being refused outright. The rule runs in that one direction only: a wide gap before a dash does not unlock the gap after it, and a hyphen inside "K-8" with a wide right gap still comes out "K- 8", exactly as the library alone would print it. A date range stays safe because its after-dash gap is below the floor, so the exception never fires there. On the sample, 652 dashes now match the answer key and none is spaced where the key keeps it glued; the 15 real documents extract exactly as they did before this exception was added.
Our coverage metric never showed this, because it drops punctuation before comparing words. A metric is only as good as what it can see.
We also kept everything else about the library's extraction: the same filters, the same grouping of characters into lines, the same run geometry, the same layout algorithm. One honest caveat: that algorithm sizes its character grid from the median width per character of the runs on the page, and a run that gains a space has one more character in the same width. So on a page where we add spaces, the grid can get slightly finer, and other code that reads column gaps from that grid to detect two-column pages sees a slightly different picture. On the 15 real documents the output is byte for byte the same, so the grid did not move there; on the dataset's documents the old output was one word per line, so there was nothing to preserve. It is still a second thing to regression-check, not something the floor guarantees.
Measuring it against an answer key
The dataset (a public resume dataset on Kaggle) came with something better than our judgment: alongside each PDF, the text of the original HTML it was exported from, in reading order with real spaces. That is an answer key, with the usual caveat that HTML text and rendered text can differ through hidden or generated content, so a miss is a signal to look, not proof of a bug.
For each document we lower-cased both texts, split them into runs of letters and digits, and compared:
- Words captured: the share of the key's words that also appear in our output, counting repeats. Recall over a bag of words.
- Words in the right order: the share of the key's words that fall inside
runs of four or more consecutive words that match our output in sequence
(Python's
difflib.SequenceMatcher).
Both ignore punctuation, which is how the dash problem above hid from them. And 100% captured means every word is present, not that the text is perfect: extra splits and ordering slips can survive it.
In our case, on 278 sampled documents:
| Before | After | |
|---|---|---|
| Words captured (median) | 9% | 100% |
| Words in the right order (median) | 2% | 97% |
The sample is half documents we had flagged as glued and half drawn at random; the "after" in-order figure is 95% on the flagged half and 100% on the other. The "before" numbers are the library's own output, the "after" numbers are the rule above at 0.08 em.
Our first cut used 0.1 em and captured 98%. Almost every word still missing followed the letter "m", with a gap of 0.092 em. Our working explanation is that the font states that glyph slightly wider than it draws it, so the measured gap comes out short; we did not take the font apart to prove it. Dropping the minimum to 0.08 em closed the gap without changing any real document; 0.06 em began to.
On the 15 real-world documents, 14 now extract byte for byte as before. The fifteenth gained three spaces it visibly has on the page, after a colon and before an opening bracket where the old extraction had glued them. Our test fixture also extracts identically, and a spec pins it.
The tool we didn't adopt
We also tried Docling (2.138, through docling-serve 1.37), an open-source document-conversion toolkit that combines its own PDF parsing with a learned layout model, in its standard pipeline. It got the spacing right on its own. It also needed about 2 GB of memory at peak and took 1 to 7 seconds per document, against a fraction of a second for our extractor. And on two of our real documents it pulled right-aligned dates away from the lines they belonged to, an error our rules didn't make.
That was the deciding detail. A general model that is right on average still broke documents that already worked, and it would have meant running a new, memory-hungry service alongside everything else. Replacing one step of the library we already used fixed the specific failure and changed nothing else. We kept the layout model as a local tool for comparison.
What we'd tell you
- Find out how your PDF text gets its spaces. If your library joins characters by a fixed gap, some generator out there places words just below it.
- Test with documents from a generator you didn't choose. Ours only surfaced when we tried a dataset made by a tool none of our users happened to use.
- Make a heuristic relative to its context. A gap, a font size or a line height means something only next to its neighbours.
- Keep the old rule as a floor when you loosen one. Then the new behaviour is a strict superset of the old, and you only need to review what it adds.
- Score against an answer key. "The text looks fine" was what we believed before this. A word-coverage number would have caught the bug on day one.
- Know what your metric cannot see. Ours dropped punctuation, so it was blind to a dash rule that was wrong on hundreds of lines.
The rule is still a heuristic tuned on one exporter and a small set of real documents, and the 278-document sample all came from that one exporter: good evidence that the fix works for it, weaker evidence about generators we have not met. What changed for good is that we now have a number for it: rerun the extractor over a set with an answer key whenever the rule or the inputs change, and a generator that breaks the rule shows up as coverage dropping, not as a user's unreadable upload.