Summaries are not reading

Ask a good model to summarise a forty-page report and it will hand you six paragraphs in a few seconds, and they will be accurate. That is not a grudging admission — it is the whole problem. If the output were bad, nobody would need this article. The output is good, which is exactly why it is worth being precise about what it is and what it is not.

Summarisation belongs in the speed-reading family and it is the newest member. The lineage runs from the hand sweep through the word-flashers and the bold-prefix schemes to a system that reads the document so you don’t have to; the field guide lays the family out generation by generation. Each one promised the same thing, which is the meaning without the minutes. The summariser is the first member to deliver on the literal promise, because it genuinely does read the whole thing.

The useful case first, because it is real and large. You cannot read every filing, every abstract, every thread, every changelog. Nobody can, and pretending otherwise is how people end up reading the first page of forty documents and the whole of none. Triage is a legitimate and necessary skill, and a machine summary is often the correct instrument for it — not a compromise, the correct instrument. A good summary tells you which document deserves your afternoon. Blinkist built a business on the observation that many people would rather have the gist of a book than nothing at all, and for a great many books that is a fair trade.

Compression is a series of judgements

Here is the argument, and it is not about quality. A summary is a compression, and every compression is a sequence of decisions about what mattered. Something read the document, decided this clause was central and that one was ornament, and discarded the ornament. When you read the summary you inherit every one of those decisions. You inherit them silently, without the option of disagreeing, because the material you would have needed in order to disagree is the material that was removed.

This is different from the usual complaint. The usual complaint is that models make things up, and they do. But a fabricated sentence is at least visible: you can check it, and sooner or later it will contradict something. The failure that should worry you is the summary that is fluent, accurate in every sentence it contains, and quietly missing the clause that mattered. Nothing in it is wrong. It is simply not the document.

Three things you give up

You cannot audit an omission

An error leaves a trace and an omission does not. If a summary states the wrong date, some part of you may notice, and the next document you read will disagree with it. If a summary leaves out the sentence explaining that the figures exclude a subsidiary, there is nothing to notice. The text reads as complete, because completeness is what a well-formed paragraph looks like from the inside. You cannot ask what was left out either, since the only honest answer is the original document — at which point you have saved nothing at all.

The interesting thing is usually the anomaly

Summarisation works by identifying what is central and representative. That is the definition of the task, and a summariser that dwelt on the odd sentence in a footnote would be judged a poor one. But in the documents where reading actually pays, the value is rarely in the central and representative part. It is in the one paragraph that doesn’t fit — the unexplained restatement, the hedge that appeared this quarter and not last, the definition that quietly changed. A system built to suppress noise will suppress that too, and it will be behaving correctly when it does.

You do not build the model

The third cost is the one people discover late. Reading is not the transfer of a text into your head; it is the process by which a representation gets assembled there, sentence by sentence, with your own objections and connections built into it as it goes. Skip the assembly and you end up able to repeat a claim but unable to reason from it. The difference stays invisible until somebody asks a question the summary did not anticipate — and then it is not subtle at all, because there is nothing underneath the sentence you memorised.

Confident wrongness deserves a paragraph rather than a chapter. Models do state things that are not in the source, and the tone does not change when they do; fluency and accuracy come apart in a way that a century of reading human prose has not trained anyone to detect. The mitigation is well known and boring — check anything load-bearing against the original. What that mitigation does not touch is the omission, because there is nothing there to check against.

An error leaves a trace. An omission leaves nothing at all.

Why summaries cannot be checked

What we are not saying

We are not opposed to summaries, and it would be convenient but dishonest for a reading app to cast the technology as a threat to reading. It isn’t one. Whatever this app competes with, it is not the summariser. It competes with not reading the document at all — with the tab you closed, the report you skimmed the first page of, the chapter you have intended to start since March. Those are the alternatives that actually claim the hours, and they win far more often than any machine does.

For a good many documents the summary is also the right and final answer. A two-hundred-page appendix of tabulated results does not deserve your eyes, and we would rather say so than sell you a way to run it past them faster. There are documents at the other end too, where neither tool applies: poetry, proofs, contracts you intend to argue with. Those want slow, recursive, sceptical reading with a pen beside you, and no amount of presentation engineering — ours very much included — is a substitute for it.

The workflow we actually recommend

The sensible arrangement is not either-or, and it takes about a minute to set up. Summarise the pile. Use the summaries as they are meant to be used, as a sorting instrument rather than as the reading. Then take the two or three documents that survived the sort and read them properly, at whatever pace leaves your comprehension intact, and test yourself afterwards rather than trusting the feeling that you understood — a feeling which, as the comprehension tax sets out, is a poor guide to whether you did.

1. Summarise everything. Ten documents, ten summaries. Treat the output as a ranking rather than as content. The only question you are asking of it is which of these deserves an hour.
2. Read the survivors end to end. Import the actual file into the app — PDF, Word, EPUB or plain text, parsed on your own machine — and read the whole of it rather than sampling it.
3. Keep the recall test on. Being tested on material retains it better than re-reading it, and the score sits next to your rate, so you can see the point at which going faster began to cost you.
4. Return to the source whenever a summary surprises you. A claim you did not expect is either the finding of the document or an artefact of the compression, and there is no way to tell which from the summary alone.

There is one habit worth adopting on top of all that, and it costs nothing. When a summary tells you something that changes your mind, go and find the sentence it came from. Not the paragraph — the sentence. Most of the time it will be there and it will say what you were told. Occasionally it will turn out to be a hedge, a projection, or a quotation of somebody the authors were disagreeing with. That is the moment you learn what a summary really is: not a smaller version of the document, but a different document about it.