Want to write a bestselling novel? Use an algorithm

It’s the multimillion pound question that publishers and writers have been pondering for decades: what makes a bestseller? Attempting to write one could certainly pay off – the highest-paid author in the world, JK Rowling, has made $95m (£70m) in the past year, and the 10 highest-paid authors in the world earned more than $310m between them, according to Forbes.

But few authors will see that kind of cash. The average annual income for a UK writer is £12,000, well below the minimum wage for a full-time job, a recent European commission report found. The Authors’ Licensing and Collecting Society says professional authors have experienced a 29% drop in income in real terms since 2005, with the top 5% earning 42% of the money made by UK writers. And the bottom 50% struggle to generate even 7% of the total income.

The promotional budget for a book is usually related to the advance, with celebrity authors attracting more than their fair share of publishers’ marketing spends, regardless of the quality of their work. And marketing budgets matter: according to the International Publishers Association, UK publishers released 173,000 titles in 2015. That’s more than 19 every hour over a year, and the highest number of new titles per million inhabitants in the world.

Spotting a book that will make money isn’t easy. Even bestselling authors such as James Patterson, Danielle Steel and Stieg Larsson had manuscripts rejected multiple times, while self-publishing sensations like Fifty Shades of Grey took the book industry by surprise.

“Everyone in publishing is always looking for the next bestseller,” says Jodie Archer – and she should know. She spent her career trying to spot bestsellers as a commissioning editor, then went to Stanford University in the US to do a PhD on the subject. “I wanted to know why the whole world seemed to be reading the same book at the same time. What is mass readership saying about the books readers want to read? And how are authors writing that commercial success, unbeknown to them, into their novels?” Instead of asking writers and readers, Archer decided to ask a computer.

“It doesn’t matter whether a book is published as literary fiction, romance, sci-fi, crime or any other genre, there are some latent features of bestseller-dom in manuscripts and these patterns are detectable by a computer algorithm,” she says. Together with Matthew Jockers, an expert on text mining, she has written an algorithm that can tell whether a manuscript will hit the New York Times bestseller list with 80% accuracy. With some authors, such as Rowling, Patterson or EL James of Fifty Shades fame, the algorithm was more than 90% certain their manuscripts would be bestsellers. “When we ran it for the first time I was stunned,” says Archer.

Luckily for would-be authors, she and Jockers decided to share what they gleaned from their algorithm in their 2016 book The Bestseller Code. “It shows how literary style, as read by a computer, has a huge influence on how people are going to react,” Archer says.

The computer also revealed how much an author’s fingerprints were in everything they wrote, she says. It can detect this in their use of nouns, the proximity of adjectives to nouns and how many commas they use.

To figure out what the bestsellers had in common, the algorithm assessed 280,000 data points in every novel Archer and Jockers fed into the computer. It found 2,800 of these data points were notable features that together could identify a potential bestseller. If you want to write a bestseller it will help to get the combination of these features right, says Archer. “You have to hit the sweet spot on character, plot, style and theme.”

Of course, most successful authors know this – and other findings seem equally obvious. For example, the algorithm found that readers of bestsellers liked shorter sentences, voice-driven narratives and less erudite vocabulary than readers of literary fiction. It also predicted authors who had worked in journalism had the greatest chance of writing a debut bestseller. “That kind of training helps you write for a popular market and means your style is often accessible and colloquial,” Archer says.

In terms of plot, a bestseller has to have an “emotional beat”. “In certain bestsellers there is an emotional high followed by a low, then another high, then another low,” Archer says. “If you get that pattern symmetrical you will keep readers turning the pages.”

What amazed Archer was how, when she used the algorithm to plot the emotional beat of Fifty Shades of Grey and The Da Vinci Code on a graph, the two plotlines were almost identical. “They were the only two out of the thousands of novels we studied that had this up and down plot at the right pace.”

The algorithm also detected the topics discussed in a book by mining the text for groups of words frequently connected with each other. It concluded that there were 500 topics (such as “love” or “lawyers and the law”) that were important for a book to be classed a bestseller, and different combinations of those topics gave readers a sense of the book’s theme.

Archer and Jockers then found that successful authors devoted 30% of their books to just one or two topics. “Less experienced writers tend to include too many topics in their first drafts so their books don’t appear to have a focus,” says Archer.

View image in fullscreen

According to the algorithm, the most popular topic was “human closeness” – when authors expressed how humans related to each other in intimate moments. This rings true for bestselling author Milly Johnson, who was one of the top three most borrowed female authors from UK public libraries last year. “The secret of my success is that readers feel as if they can walk down the road and meet my characters,” she says. “They can relate to them. They want to be friends with them.”

It also holds true for Robert Muchamore, the bestselling author of the Cherub spy novels for teens. “I thought kids would respond to my books because of the missions and spy stuff,” he says. “But after my first book came out and fans started writing to me, I discovered that what kids really liked was the intrigue of who was getting off with who and the high jinks on Cherub campus where all the agents lived. The books that followed were altered by the feedback I got from fans.”

Human relationships have to be at the core of a bestseller, he adds. “If your reader doesn’t care about the characters, they’ll feel emotionally flat at the end.”

Muchamore, who has sold more than 12m books, puts his success down to timing and choice of topic. “There was a gap in the market for teenage spy novels and my book was in submission when Anthony Horowitz’s spy series for a younger audience took off.”

Johnson spotted a similar gap for romantic comedies about Yorkshire women in their late thirties. But she doesn’t believe there’s a formula you can follow. “If you set out to write a novel to make money, you’ll never do it. You have to write the book inside you.”

Muchamore is equally sceptical about whether an algorithm can produce a bestselling novel. “The proof will be in the pudding,” he says, adding that he is planning to buy a copy of The Bestseller Code. “It sounds interesting.”

Archer says many authors have asked her how they can improve their manuscripts. So many, in fact, that she and Jockers are setting up an agency later this year that will allow writers to access their algorithm. For an as-yet undisclosed fee they will give writers a report on how the algorithm has assessed the likelihood of their manuscript getting on the NYT bestseller list, and time with them to find out how they can improve their chances.

“They will get a bird’s eye view of the emotional action of their plotlines and a spatial understanding of the book – for example, where and how much different themes are showing up,” says Archer. “That will help authors see the bigger picture of their book. It’s hard to do that with your brain.”