An Experiment in Asking LLMs to Edit Public Domain Books

Sep 05, 2026

I created the Public Domain Diffs repository with two ideas in mind:

First, that LLMs are good at editing. Not perfect at editing, but neither are books perfect. So in theory, a sufficiently smart LLM could improve upon the published works of respected authors. (Where I get to decide what is and isn't an improvement, on account of my having good taste.)

Second, that the type of edits that current LLMs are best suited for — local, targeted changes — are also exactly the type of changes that diff viewers are built for.

Together, the plan became to combine the smartness of LLMs with the tooling of version control: I'd create a repository that had full-text versions of famous books as its root master branch with LLM-edited child branches off of it:

Each LLM's edit of each book would be captured in an individual commit to its LLM branch, making the LLMs' edits easy (in theory) to examine:

Provided with ten famous(-ish) public domain texts: Could the LLMs find areas for prose improvements? Could they make those good edits while avoiding making bad or superfluous edits?

(For crappy source texts, we already know the answers to these questions are "yes" and "mostly yes". Hence why we want the more interesting challenge of running against more highly regarded works. The improvement potential should still be there, even if only marginally so.)

This diff-based version controlled approach seemed the most efficient way to find out, with the added bonus of being able to compare the relative editing intelligence of competing LLMs.

The Setup

I knew from previous book-processing experiences that the LLMs were not yet good enough to process novel-length texts intelligently all in one go — the context size degradation would be too severe.

Instead, the plan would be to just break the books apart into chapter-length chunks, using a short looping script that fed the individual chapters into the LLM APIs with this prompt:

You are going to be given the task of editing a chapter from a famous public domain book, with the goal of improving its prose. Your edits can take any semantic form — adding, removing, restructuring, etc — using your own best judgment on what what prose changes have the potential to improve the existing text's quality. No author is perfect over the course of an entire book-length text, so I expect you to be able to identify some areas for enhancement.

At the same time, you should not over-edit — don't make lateral changes that lack a clear purpose. I expect you to be able to tell the difference between good vs mediocre prose and therefore when editing is unnecessary vs desirable. Also remember that the goal is to improve specifically the prose — not to change the overall book's identity. Don't butcher the text's plot, characters, or intended authorial voice — keep the scope of the editing targeted and local.

For this task, bear in mind that you'll only be provided with the text one chapter at-a-time. This shouldn't cause you any trouble in keeping track of the book's plot, which you already know, but it does mean that you won't be able to see how your parallel editing of the other chapters is going. To keep the scope of editing consistent across chapters then, you'll have to figure out your own internal editing-density Schelling point. (Left again to your own best judgment, since you know your own LLM brain better than I.)

The book you'll be editing is <book> by <author>.

Output the full text of the chapter back out to make it easy for me to diff the results against the original chapter to see what edits you've made. (And output juts the chapter text, starting with the chapter header.)

In the prompt, I tried to preempt several problems I foresaw that might hurt the editing:

If the LLMs were at all competent, they would be able to avoid all of these pitfalls and also know to scale their editing inversely with the "tightness" of the various source texts.

That is, I had a specific least-editable-to-most-editable ordering in mind that I wanted to see borne out in the relative diff densities between the ten books:

This set of books represents a good range in input quality (noting that books can get significantly worse yet), so I had high hopes for the experiment.

Unfortunately:

The Problems

With the prompt, the books, and the predictions all set up, I went ahead with running the loop and committing the results, and then eagerly began my examination of the diffs. And I immediately ran into most of the issues I'd thought I had preempted, plus several additional ones.

The diffs were, in many places, a nightmare to read. The LLMs were fond of splitting and merging paragraphs: fine changes to have made in theory, except that they wrecked the readability of the diffs in all three viewers I tried. See kdiff:

And Rider:

And GitHub:

Additionally, the LLMs would often quit halfway through their edits, inexplicably, leaving huge chunks of chapters missing (a problem which reprompting did not consistently solve, for some reason):

The diff density was also highly inconsistent between chapters, varying from over-edited to insanely over-edited.

Despite my "clever" prompting's best efforts, the diffs were full of pointless changes: lateral word choice replacements, unnecessary comma adjustments, unnecessary spelling adjustments (both British-to-American and American-to-British), and actively detrimental non-ASCII character deletions against hair spaces and so forth.

So my initial plan of carefully combing through all the diffs rapidly deteriorated into skimming the diffs.

And my secondary plan of redoing the whole process with better prompting got delayed and delayed and derailed by other priorities, such that now — five weeks later — I decided enough is enough, and that I need to just go ahead and write up the results I did find, with these acknowledgments that the process ended up being badly flawed.

But as unsatisfying and incomplete as the results and their examinations are to me, there were still at least some interesting patterns of changes I managed to spot:

The Results

In displaying the edits, I'll show the before-and-after in pairs: original book text on top and LLM-edited text on bottom.

This is the source text.
This is the edited text.

I failed to catalog the results well enough to include the book and LLM editor that each change corresponds to — a limitation that ended up being not too terribly meaningful anyway, since the differences in behavior across the different books and between the two LLMs all ended up being far less interesting than expected.

A follow-up experiment would need to do this better, though even for this initial one, the information is technically accessible via cloning PublicDomainDiffs and using a combination of grep and git log -S.

In any case, working with what information is readily accessible now:

Lateral Edits

The vast majority of the LLMs' edits were nothing changes that have no real impact on the reading experience:

fixed a pillow under her head
fluffed a pillow under her head
“Foolish child! foolish child!”
“Foolish child! Foolish child!”
Dundy asked: “Well, what happened then?”
“Well, what happened then?” Dundy asked.

A better version of this experiment would need to somehow better impress upon the LLMs the importance of not flooding the diffs with meaningless changes like these.

In a few cases, the lateral changes were somewhat more interesting in that they represented the LLMs insisting on being technically correct:

The old Guiacum test was very clumsy and uncertain.
The old guaiacum test was very clumsy and uncertain.
the unemotional Indians, journeying in with their pelties
the unemotional Indians, journeying in with their peltries
Spaniards strangled Edward Mansveldt on the garotte
Spaniards strangled Edward Mansveldt on the garrote

Though I don't give the LLMs much credit for these micro fact-checks, since they're the type of fixes that no one would ever notice.

The only really interesting type of change that fell under the "lateral" umbrella involved the LLMs' tendency to try to clarify words potentially confusing to the modern reader, such as "flags" or "theatre" or "arcade" or "leaders" or "transit":

a quadrangle paved with flags and lined by sordid dwellings
a quadrangle paved with flagstones and lined by sordid dwellings
as he crossed the theatre
as he crossed the anatomical theatre
They were carrying tables out and setting them up under the arcade.
They were carrying tables out and setting them up under the arches.
some had leaders upon it
some had leading articles upon it
revolved it slowly, tracing, like the leg of transit, a thin red circle
revolved it slowly, tracing, like the leg of a compass, a thin red circle

None of these represent an actual information change for the reader who already correctly understands the words, but for the reader that doesn't, I suppose this kind of editing is… okay? It's at least not something I would have ever thought to do, which makes it somewhat interesting.

Bad Edits

There were many bad edits — more than there were good edits — a big disappointment. Though there were at least a variety of patterns in the bad edits, making them somewhat interesting.

First, there were the cases where the LLMs couldn't help themselves but to overcorrect grammar:

“A good, swift sailor,” Henry observed
“She looks a good, swift sailor,” Henry observed
Marvelous plan! Genius of an officer!
A marvelous plan! The genius of an officer!
he’d of got me sure.
he’d have got me sure.
“Taken who to prison?”
“Taken whom to prison?”
“Your frients haff gone upstairs,”
“Your friends have gone upstairs,”

These represent a subclass of the problem of the voice of the LLM trying to insert itself over the author, either via weakening the word choice or just inserting a distracting internet-y style where it doesn't belong:

Spade was stupid. “How what?”
Spade played dumb. “How what?”
His reading had been tremendous.
He read voraciously.
The youth loitered with other loiterers
The youth mingled with the other loiterers
Lady Narborough hit him with her fan.
Lady Narborough tapped him with her fan.
he had studied and considered what navigation there was in books
he had devoured what books on navigation he could find
Henry Morgan was aroused.
Henry Morgan flushed with anger.
His face was stupid in its calmness.
His face was stolid in its calmness.
He was not prepared for this change of idea. He was staggered at such a revolt against his preconceptions.
He was staggered by this subversion of his expectations.

Also included were cases of useless bowdlerizing:

they had to travel in Indian file
they had to travel in single file
Strong, glistening blacks pulled at the oars of the canoe
Strong, glistening men pulled at the oars of the canoe

And actively rewriting sentences to be stupider:

He felt that that was to play his enemy’s game
He knew that dying would be playing right into his enemies' hands
He spoke jocosely.
He spoke with a grim, forced lightness.
The day-coach—he was penniless now—was hot.
The day-coach was stiflingly hot, and he was penniless now.
“Don’t talk so much, old sport,” commanded Gatsby. “Play!”
“Don’t talk so much, old sport,” commanded Gatsby gently. “Just play.”

But most egregious of all were the cases where the LLM thought it had any authority to override literature's identity, whether that be Austen's iconic jank commas or Watson's famous ejaculations:

It is a truth universally acknowledged, that a single man in possession of a good fortune, must be in want of a wife.
It is a truth universally acknowledged that a single man in possession of a good fortune must be in want of a wife.
“Wonderful!” I ejaculated.
“Wonderful!” I exclaimed.

And then lastly there were some scattered cases of the LLMs just making strict mistakes in their transcriptions, their probabilistic nature showing through:

their pride held them inexorably apart
their pride held them inexroxably apart
“Tell him I cannot see anyone,” it said complainingly.
“Tell him I cannot see anyone,” it said complain ingly.

Overall, lots of lots of bad.

However. More bad than good can be acceptable — even if not desirable — when the results can be human-curated afterwards. If the LLMs had spit out loads of garbage but wedged in a sufficient number of banger edits, the bad could be acceptable.

Instead:

Good Edits

They were few and far between. And even where they were present, the improvements themselves were rarely more than marginal.

First off, there were the basic prose simplifications:

“You appear to be astonished,” he said, smiling at my expression of surprise.
“You appear to be astonished,” he said, smiling at my expression.
The Boots volunteered to show me the room: it was on the second floor, and there was a small corridor leading up to it.
The Boots volunteered to show me the room. It was on the second floor, down a small corridor.
He took no particular notice of him, beyond thinking in his own mind that it was early for him to be at work.
He took no particular notice of him, beyond thinking it was early for him to be at work.
The young fellow seemed pleased at the suggestion, and his dark eyes sparkled with pleasure.
His dark eyes sparkled at the suggestion.
a singularly simious and apelike appearance
a singularly simian appearance
lest something which fell from their lips might be misconstrued, and bring down a swift retribution upon them
lest a misconstrued word bring down a swift retribution upon them
The afternoon passed slowly, procession-like with an end of evening, and on the heels of the evening an excited wind rushed out
The afternoon passed slowly, a solemn procession toward evening. On its heels, an excited wind rushed out,
a degree in science or any other recognized portal which would give him an entrance into the learned world
a degree in science or any other recognized avenue into the learned world
“No,” answered his mother with a placid expression in her face.
“No,” answered his mother with a placid expression.

And there were some word choice improvements:

It was a foggy, cloudy morning, and a dun-coloured veil hung over the housetops, looking like the reflection of the mud-coloured streets beneath.
It was a foggy, overcast morning, and a dun-coloured veil hung over the housetops, mirroring the mud-coloured streets beneath.
with sad brown eyes like a dog’s eyes
with sad brown eyes like a hound's
Mary even thought she saw him wink his eyes as if to wink tears away.
Mary even thought she saw him blink as if to keep tears away.
Mary hesitated about two minutes and then boldly spoke the truth.
Mary hesitated for a long moment, and then boldly spoke the truth.
“How is it now?” he answered anxiously
“How is it now?” he asked anxiously

And there was even the occasional useful grammatical fix:

“And who do you expect will answer this advertisement.”
“And who do you expect will answer this advertisement?”

But none of these changes represent any kind of great value that would have made this whole process worth it.

There were perhaps only three cases where the LLMs managed to really impress me with their ability to improve the voice of the author, either through better word choice or better mode of description:

his eyes had assumed the vacant, lacklustre expression which showed mental abstraction.
his eyes had assumed the vacant, lacklustre expression that betokened mental abstraction.
This play of claque to a simpleton was a harsh tax on the convictions of Mother Morgan.
This pantomime of awe was a harsh tax on the convictions of Mother Morgan.
She had sprung away from him.
She sprang away from him.

With the first of the three coming from A Study in Scarlet and the next two coming from Cup of Gold. So ultimately the predicted editing value ordering was correct — technically — accepting the eight-way tie at zero in last place.

Lessons Learned?

This process updated me against the practicality of using current-generation LLMs to improve published, well-authored books.

It also updated me against the value potential of these kind of small-scale edits in general for those same books — the changes were ultimately far smaller than the changes you'd see between e.g., different translations of the same book into another language, which usually go unnoticed anyway.

But neither result changes the fact that A Study in Scarlet and Cup of Gold (and perhaps a few of the others, to a lesser extent) are still flawed books that could be improved — even if those improvements would need to occur on either a larger scale, or via a better prompted (or just smarter) LLM.

Therefore, I would like to see the experiment repeated, but it perhaps should not be anyone's highest priority to do so.

(Womp womp?)