Grammy-garlanded pop singers, Oscar-scooping film stars and Olivier award-winning thesps can have an ambivalence about success that surprises us mere mortals. But the American record producer Rick Rubin’s response when the rap-rock trio Beastie Boys’ debut album, Licensed to Ill, which he co-produced, topped the US charts in 1986 takes some beating. “I remember one of the people we worked with called me,” Rubin, 59, recalls, “and said, ‘You have the No 1 album in the country, how does that feel?’ And I said, ‘I’ve never been more unhappy in my life.’”

Licensed to Ill went on to sell more than ten million copies in the US, a remarkable baptism for Rubin, who, in the decades since, has also produced or co-produced multimillion-selling monsters by Public Enemy, Slayer, Red Hot Chili Peppers, the Cult, System of a Down, Shakira, Johnny Cash, Tom Petty, Jay-Z, Adele and more.

In person the Long Islander calls to mind both Karl Marx and Michael Palin’s Boring Prophet in Life of Brian, his extravagant beard and guru hair giving him the air of a swami. Bach tinkles in the background in the vast west London drawing room we sit down in. It’s hard to square the man before me with the young tyro who, while still at university in New York, co-founded the trailblazing hip-hop label Def Jam and set the charts on fire. That was then, Rubin says. He has mellowed now.

“I have a family and a great life outside of just the art. Earlier in life it was only the art. I can remember working with an artist and Christmas came and I couldn’t understand why they didn’t work on Christmas. Like, ‘We have work to do. It’s just another day!’ I literally worked seven days a week for years, never had a vacation, and didn’t want one. I didn’t think in those terms; my dream was to make things. For probably the first 20 years of my career I had no experiences other than being in a room without windows, making music.”

In person Rick Rubin calls to mind both Karl Marx and Michael Palin’s Boring Prophet in Life of Brian, his extravagant beard and guru hair giving him the air of a swami.

One anecdote in particular illustrates just how full-on Rubin’s working practices once were, to the exclusion of everything else. “When we were recording Jay-Z’s 99 Problems at the studio in my Hollywood home, upstairs in the house there was an event for Tibet hosted by Mike D of Beastie Boys. The Dalai Lama may well have been up there. It’s long enough ago now that I can’t remember the details. Johnny Cash may have been staying in the house as well. It’s all a bit of a blur. My focus was on what was happening in the moment in the studio.”

Rubin is in London to prepare for the publication of his first book, The Creative Act: A Way of Being. It’s not a spill-all memoir by any measure, rather a meditative and absorbing mix of how-to guide and note-to-self. There are chapters — among them “Seeds,” “Breaking the Sameness” and “Surrounding the Lightning Bolt”— about collaboration, unlocking the creative process and staying true to the idea that, as Rubin puts it, “the art comes first”.

Among many things Rubin is famous for saying no to Adele. He had worked with the singer at his Shangri-la studio in Malibu on her 33 million-selling second album, 21, and was invited to listen to the work in progress on the follow-up. Not liking what he heard, he told Adele that the songs didn’t sound authentic and advised her to start again. Which she duly did. Speaking about the encounter after the successful release of 25, however, she admitted she had been deeply shocked by Rubin’s verdict at the time. “When he said it, I couldn’t work out if I was, like, devastated, going to cry my eyes out,” she recalled. “And then I just said, ‘I don’t really believe myself right now, so I’m not surprised you f***ing said that.’”

Rubin defends this approach robustly — both with Adele and other artists he has worked with. “The role [of a producer] is to keep the trains running and I’m not that. The normal thing in this job is to make sure everything’s crossed off and turned in on time, but it’s the furthest from what we do. My interest is only in making the best thing we can possibly make. Even for the people who care about the dates and get mad when the dates are missed, if it’s the best that it can be, it will serve everyone, it’ll be better for the label, the management, for everyone, even though they may not see it that way.”

“The role [of a producer] is to keep the trains running and I’m not that.... My interest is only in making the best thing we can possibly make.”

Rubin spoke movingly about his struggles with depression when he appeared on Desert Island Discs in October. One of his song choices was the Beatles’ Across the Universe, which contains John Lennon’s iconic line “Nothing’s going to change my world”. What do you think of when you hear that, Rubin asks me. I tell him that to me it means Lennon has his core and that nothing — Beatlemania, tensions in the band, the vilification he and Yoko received — can threaten that. It’s telling, and a little heartbreaking, that Rubin detects something quite different. His comment tracks back to his reaction to Beastie Boys’ success.

“When I hear that I hear it as a tragic line. What it makes me feel is: nothing’s going to make it better. That’s my experience of life. Now I’m fine, I’m good, but my default setting is moody.

“I still don’t really know how to negotiate life. The creative situation is my place of comfort, and anxiety. I don’t know where real life begins and ends. But the fact that I can acknowledge that we don’t know what’s going on makes me closer, I think, to working it out than someone who goes, ‘It’s all spelt out.’”

Rubin spent five years as co-president of Columbia Records between 2007 and 2011, with mixed results. The experience confirmed his feelings about art v commerce, and he retreated from the shark-infested waters of the music business back to Shangri-la and his work as, he says, an enabler rather than a cajoler. He is one of the most successful producers of all time; whether he feels successful, whether the acclaim grants him peace, is another matter.

The Creative Act, full of advice and aphorisms, when others in Rubin’s position might simply have name-dropped and basked in self-glory, is, he says, realizing it, he admits, for the first time as we talk, “in some ways written to myself. I have said that I wish I’d had this book when I was young, but now I’m realizing that’s really who it’s written for. I can read it in proof now and be like, ‘Yes, that’s how it is,’ but sometimes I’m like, ‘Wow, I didn’t know I knew that.’”

Rubin isn’t referring to his uncertainty about whether the Dalai Lama once called round — but he might as well be. Rooms without windows are a sanctuary, yes. But they can also be a trap.

Dan Cairns is a music editor and features writer for The Sunday Times