The stagnation blues

In 1987, Bob Dylan realised that he was a dead man walking. Motivated more by cash than by conviction, he had committed to two world tours, one with Tom Petty and the Heartbreakers and one with the Grateful Dead, and he had played enough dates to know that the jig was up.

"There was a missing person inside of myself and I needed to find him," he remembered in his memoir Chronicles: Volume One. "Now and again, I did try a few times, tried hard to force it… But it was no use. I felt done for, an empty, burned-out wreck."

Dylan's last great album, Desire, was over a decade old while his most recent records, Knocked Out Loaded and the yet-to-be-released Down in the Groove, were his worst yet: grimly professional and dead inside. Everything that had once seemed simple and instinctive now felt alien to him. As a performer, he was so estranged from his own songs that playing them felt like "carrying a package of heavy rotting meat". As a writer, he was "an old actor fumbling in garbage cans outside the theatre of past triumphs". He was boring both himself and his audience and he could see no way out apart from a dignified retreat. "I had no connection to any inspiration. Whatever was there to begin with had all vanished and shrunk… It wasn't my moment of history any more. There was a hollow singing in my heart and I couldn't wait to retire and fold the tent."

The bleakness of Dylan's account in Chronicles is relentless and unqualified. For several terrible months, perhaps the greatest songwriting talent of his generation felt in his bones that he had been exiled from himself with no way home.

Every music fan knows that creative stagnation is a common fate. As Sick Boy succinctly phrased it in Irvine Welsh's Trainspotting, "At one time, you've got it, and then you lose it, and it's gone for ever." Why you lose your songwriting gift, however, is as mysterious as how you get it in the first place. Bono recently asked, "Why have the most prolific imaginations ceased to be when they don't in literature, and they don't in film, and they don't in painting, and they don't in poetry? What's that about?" He sighed. "I don't have the answer.'

Few creative people can manage the painful honesty that Dylan did. Most artists convince themselves, as they must, that their new album is one of their best and critics and fans generously collude in reassuring them, perhaps by using that tired phrase "return to form". As a term from the sporting world, where form is arithmetically measurable, it is virtually meaningless with regard to art. When we say someone's new album is a return to form, we mean it is less lousy than the previous couple, not that it's worthy to stand with the artist's early triumphs. We know this, and they know this, but it feels cruel to spell it out.

Most artists learn to live with their waning powers. Some are happy to put out what is essentially the same album again and again because age brings certain compensations. Often, older musicians are healthier, wealthier and more well-balanced, with more harmonious relationships at home and work. As long as they can still play their classic songs to adoring audiences, they needn't scourge themselves for their inability to produce comparable new ones. The further up the scale you go, the truer this is. The likes of the Rolling Stones and Depeche Mode long ago became such lucrative megabands on the touring circuit that even at their lowest creative ebb they were too big to fail.

Real stagnation, on the other hand, is a profound existential crisis. You lose touch with music and thus with yourself. The past taunts you and the future blanks you. You know there's a problem but you don't know how to fix it. As Dylan writes, "There didn't seem to be any formula." Stagnation can mean coasting on autopilot, living on past glories, but it can also mean desperately trying different strategies without success.

The 80s was the harshest decade in that respect, as the likes of Neil Young, David Bowie, Lou Reed and Paul McCartney joined Dylan in fumbling down a series of blind alleys. Technological upheaval made the 80s particularly challenging, as baby-boomer stars struggled to work out what to do (and what not to do) with MTV and synthesisers. But it's no coincidence that all these artists were aged between 35 and 40 when that decade dawned; in the throes of the classic midlife crisis years when people in many walks of life are beginning to wonder if their best work is behind them. Young's description of his mid-80s nadir sounds like a nightmarish loss of self. "I felt disconnected – I felt it too. When I played my music, I went, 'What's going on? Where am I? Where is it?'"

Young compared that feeling to drowning and struggling to come up for air. Tom Waits saw himself as a wind-up toy hitting a wall and still whirring fruitlessly. In Chronicles, Dylan describes himself as a burnt-out match, a car that won't start, a house with all the windows boarded up.

They all came back, despite the odds. But how?

One night in the dawn of the 80s, Tom Waits had a bad dream. He was in a Salvation Army store, rummaging through some cultural flotsam and jetsam, when he came across a stack of dog-eared vinyl LPs. One of them was his. "The sleeve stared at him reproachfully, and he knew something had to change," writes Waits's biographer, Barney Hoskyns. "He didn't want to be a has-been, a seventies bargain-bin relic. He had to create something unique, 'something you'd want to keep'."

To outside eyes, Waits appeared to be doing okay. He had established a cult following with a thin but compelling persona: a maudlin, older-than-his-years barfly with a hat, a piano and a glass of Scotch; an Edward Hopper nighthawk made flesh; 'a downwardly mobile escapist', according to critic Robert Christgau. That guy had made six solid-to-great albums for Asylum Records, squeezing a lot of colour out of a limited palette. But in private, he felt like the exhausted star of his own long-running sketch show, trapped in his old material. "I'd nailed one foot to the floor and kept going in circles, making the same record," he said years later.

To remove the nail, Waits needed help and he was lucky enough to find it at just the right time. In 1980, he was working on the soundtrack to One From the Heart, Francis Ford Coppola's Las Vegas-set musical drama, and got to know Kathleen Brennan, a script analyst working for the director. They fell in love and were soon married. Brennan introduced her new husband to a new world of music and encouraged him to push the extremes of beauty and ugliness in his work. "I was like an old man, stuck in my ways," he said in 2004. "Because my music up to that point was still in the box, I was still in the box; hadn't unwrapped myself yet."

For 1983's Swordfishtrombones, Waits changed everything. He sacked his old band and manager, left Asylum and moved to New York. He incorporated new influences, notably Captain Beefheart and maverick composer Harry Partch, and new instruments, including calliope, marimba and squeeze drums. He based lyrics on images rather than moods or characters. He found an abrasively gruff new singing voice. But Waits didn't feel like he was changing himself; more that he was becoming himself. No longer spinning his wheels against the walls of his adopted persona, he was free.

Swordfishtrombones gave Waits the career and reputation he has to this day and became a case study in how to bust yourself out of a rut. "I think I was envious," Elvis Costello told Barney Hoskyns. "Not so much of the music but of [Waits's] ability to rewrite himself out of the corner he appeared to have backed himself into."

In some ways, this reinvention was a violent act. It certainly wounded some of his former bandmates, who felt cast aside. Waits's method suggests that you cannot ease yourself out of stagnation; you need a jolt. And perhaps you need to leave yourself no way back. "It's very hard to stop doing things you're used to doing," he has said. "You almost have to dismantle yourself and scatter it all around and then put a blindfold on and put it back together so that you avoid all habits."

It's worth wondering what would have happened if Waits hadn't met Brennan, who served as co-writer and producer on subsequent albums. He has said, "I knew that I wanted to change but I didn't really know how to do it." Wanting change is the easy part; making it happen is the challenge. Even with solo artists, the myth of auteurship is misleading, because almost nobody truly works alone. For Waits, marrying Brennan was the catalyst for making other major decisions that altered the shape of his life and allowed new music to flow.

David Bowie was experiencing a brutal midlife crisis when he made his worst albums, Tonight (1984) and Never Let Me Down (1987). "I knew David wanted to do a different kind of music but he just didn't know how," remembered his guitarist Alomar, unwittingly echoing Waits. Bowie overcame this paralysis only after he met his second wife, Iman, and his new guitarist, Reeves Gabrels. He may not have done his finest work in the 90s but at least he was hungry and energetic again and he never made another record as indefensibly moribund as those two.

Frustrated artists are always looking for that new producer or collaborator who can shake the foundations and make them anew. One man in particular understands this very well.

In 1988, producer Rick Rubin left Def Jam, where he had changed the sound of hip-hop with Run-DMC, LL Cool J and the Beastie Boys, to set up his own rock-focused label, Def American. He recently recalled his next move on the website Genius. "For the most part, I had been producing young artists, baby bands," he wrote. "I thought it would be interesting to find an old legendary artist who wasn't doing good work and maybe do the same kind of stuff we were doing with the young acts, with the same care and attention, like, 'This is the best we can do' with someone old. That was the whole idea. I started thinking of who to cast in that role, and the first person I thought of was Johnny Cash."

As he entered his seventh decade, Cash was experiencing a slump so deep and persistent that it seemed very likely it would never end. He hadn't managed a mainstream hit since 1976 and Columbia Records had given up on him years before they actually dropped him. The industry treated him like yesterday's news, so he began to think of himself that way. He later said, "I let the music get away from me. Because I didn't care."

Rubin arranged to meet the country singer backstage after a concert in Orange County, California. "In his mind, he was pretty much done," Rubin explained. "He was touring and playing oldies, but I don't think he thought of himself as a recording artist. I think the only reason he agreed to do it was because he felt like he had nothing to lose."

Having 'nothing to lose' is a common trope. That's how Dylan felt in 1987. The Rolling Stones, however, have always had a great deal to lose and therefore little incentive to dismantle themselves. Perhaps, like the cliche of an alcoholic entering rehab, artists need to hit rock bottom in order to dramatically reinvent themselves.

Cash's daughter Rosanne described Rubin as an 'angel' who entered her father's life and told him, "I know who Johnny Cash is and that's what I want on record — just the essential you." Rubin is not a distinctive sonic architect, like Phil Spector or Timbaland, who leaves his fingerprints on everything he touches. Nor is he an ingenious disrupter like Brian Eno, who spurs artists to outdo themselves by using habit-breaking experiments and counterintuitive strategies. Rubin's skill as a producer is to peel away all distractions, scrape off the barnacles of convention, and encourage artists to become their best selves once again.

Let's return to Rubin's account for Genius. "Something that I learned through the process is that when artists have done it for a long time, a certain pattern takes over their lives. They're on the road, and then there's a window where they can make a record… Not a lot of care goes into it. My job is often just breaking that pattern. We're going to take as long as it takes, like it's the most important thing in the world, and make the best record of your life. When I said that to Johnny, he looked at me like I was insane. It was just such a foreign concept that he could do something great."

Another important thing Rubin told Cash was that the past didn't matter. The weight of history does as much harm to veteran artists as the pressure to keep up with current trends. After Bono met Bob Dylan in 1987, he commented: "He's very hung up on actually being Bob Dylan. He feels he's trapped in the past and, in a way, he is." So Rubin told Cash to forget the idea of Johnny Cash the icon, and just find a way to be Johnny Cash, a man in his 60s, trying again. "All he needed to do was make great music that reflected who he was at that moment. He didn't need to compete with himself."

Rubin reminded Cash of his first mentor, back in the 50s: Sam Phillips of Sun Records. Both of them wanted to strip Cash down to his barest essentials. The album that the two men recorded in Cash's living room, American Recordings (1994), restored the singer to living legend status and their working relationship continued until his death in 2003. The project remains the platonic ideal of what can be achieved by an iconic artist and a young producer.

If there is, as Dylan said, no formula for a creative comeback, there are at least some common factors. Feeling so lost and depressed that you have nothing to lose. Finding patient and enthusiastic new collaborators who believe in you. Exploring new influences, instrumentation and working methods. Shedding the burden of your own back catalogue. Cutting ties, perhaps violently, with your recent past. And, hardest of all, persuading yourself that greatness is still within your reach.

For Bob Dylan, salvation came during a concert in a gale in Locarno, Switzerland on October 5, 1987. He froze up. The hacky, professional tricks that had gotten him through the tour so far stopped working. He had 'nothing to lose'. And in that moment of panic he broke through to somewhere new.

"I just did it automatically out of thin air, cast my own spell to drive out the devil," he writes in Chronicles. "Instantly, it was like a thoroughbred had charged through the gates. Everything came back, and it came back in multidimension. Even I came back and it left me kind of shaky."

Dylanologists, armed with bootlegs and bullish confidence in their own taste, argue that the Locarno concert was nothing special and the turnaround happened at another show but Dylan certainly experienced an epiphany somewhere on that tour. His account of that moment is somewhat mystical but he set about making concrete changes to his life like a man who had just received a reprieve from death row. One key decision was to break the usual touring cycle. In 1988, he embarked on what became known as The Never Ending Tour, during which his classic songs would be bent, battered and reimagined every night, so that they would never again feel dead in his hands. Another was to secure a new producer to midwife the songs that were suddenly pouring out of him. Over dinner one night, Bono suggested that Dylan use U2's producer Daniel Lanois and phoned him, on the spot, to get the two men talking.

Dylan met Lanois in New Orleans and the producer told him what Rubin had told Johnny Cash: "You can make a great record, you know, if you really want to." There was no rush. Reconvening in New Orleans the following spring, they slowly felt their way into the songs until they had a record they could both be proud of. Like Neil Young's Freedom or Lou Reed's New York, Oh Mercy saw out the benighted 80s on a relative high.

Dylan's comeback wasn't quick. His next album, Under the Red Sky, was weak and it took two albums of folksong cover versions to reopen the songwriting well that fertilised his second Lanois collaboration, 1997’s Time Out of Mind. He never claimed the songs on Oh Mercy were as strong as the ones he wrote back when songwriting felt like "seeing into metal and making it melt". But the worst was behind him. The waters that had stagnated were flowing again.

After the Locarno show, Dylan had been struck by a life-saving revelation that there was, after all, a way out of the ditch he'd been stuck in — a way to get back at least some of what he thought was gone forever. "I saw that instead of being stranded somewhere at the end of the story, I was actually in the prelude to the beginning of another one."

Perhaps this is the most important thing a musician in a rut needs to do: believe that a future is possible.