There is always something new to learn in biography, and in my 40 years in the field, I have made some crashing mistakes. The problem is that one only tends to realize how badly one has misjudged a person when it’s already too late.

As an example, there is the best-selling biography I wrote in 1998 about Stephen Sondheim. Those were the days when publishers—in my case, Alfred A. Knopf—sent successful authors across the country from east to west, ending in San Francisco. There I was met at the airport by an enormous pink limousine, replete with all the comforts of home—cushions, telephones, newspapers, magazines, and its own miniature refrigerator containing Dom Pérignon and ice cubes.

After tucking me in with a lavish foot warmer doing its best to look like leopardskin, the elegantly attired chauffeur, straight out of Sunset Boulevard, politely inquired if I was a movie star. I replied hastily, “Oh, no, I’m just a writer.”

From left, Sondheim, Arthur Laurents, Hal Prince, Robert E. Griffith, Leonard Bernstein, and Jerome Robbins in rehearsal for the stage production of West Side Story, 1957.

On the surface, I had made a great coup. Sondheim was famously reclusive, hypersensitive to slights real or imagined, and un-gettable, so it was said. While I was writing a biography of Leonard Bernstein, his name was bound to come up. I remember our hour-long conversation that resolved something in my mind. In the course of explaining Bernstein’s later works, which rapidly fell from favor, Sondheim casually explained what was wrong with them. Bernstein, he said, had a bad case of “important-itis.” That made so much sense I could have kissed him. So after I finished Bernstein, I asked Sondheim if I could write about him. He politely declined.

My Bernstein book was well received, even though it appeared more or less simultaneously with the “official” life by Humphrey Burton, who had exclusive access and better connections. I did not realize just then that I also had connections, through someone whose advice Sondheim usually took.

With decades of experience behind him, and endless advice to impart, Arthur Laurents, the playwright, first got to know Sondheim when they were working together on West Side Story. Laurents realized the young lyricist was a “comer.”

Laurents saw himself as a talent spotter. He had helped to nurture the stellar career of Barbra Streisand and also liked the look of me—after reading my book on Bernstein, he wrote, “This is Lenny. That most fascinating paradox who eluded everyone has been movingly captured by Meryle Secrest.” So he went to work on Sondheim. Then he suggested I ask him again.

This time, he said yes.

Sondheim at the opening night of Cabaret, 1966.

Our interviews took place in Sondheim’s 19th-century Turtle Bay town house. A tiny iron railing separated the house from the street, supporting boxes of impatiens leaning rakishly toward the door. A single, shallow step, and one had reached the doorbell. It was answered by a sort of cook-cum-butler, wearing a clipped mustache and a neutral expression. One entered a low hallway, its walls giving way to a curving staircase, and was greeted by a reception committee comprising a huge black poodle and a tiny, ancient dog, said to be ailing.

Beyond that was a large sitting room flanked by couches in tobacco tones and interesting small sculptures on coffee tables. I had already taken note of the omnipresent, framed puzzles of uncertain origin and vintage, now omnipresent on the walls. I had missed the stage-set aspect of the room itself. If this was Bohemia, it was distinctly of the upmarket kind.

I was at once offered a drink—“We have everything”—and in the weeks and months to come would be served tea in Royal Doulton porcelain cups with delicately wrought handles.

Then Sondheim arrived. He was drinking straight vodka from a small glass and settled back expansively on a sofa, wearing rumpled pants and a bluish-green T-shirt the sleeve of which, I noticed, was twisted and which he had not bothered to unwind.

He looked older than I remembered. I tried not to look at him too directly, fearing to make him feel uncomfortable. He seemed at ease, yet paradoxically on guard; he had a way of closing his eyes when he talked and then judging his effect on you with the briefest of looks. I was taping him, and I discovered he was also taping me, after a small tape recorder fell out of his pocket.

Then I made the mistake of moving closer, to a chair at a right angle to the sofa he was sitting on. Midsentence, he shot up like a startled fawn and moved to the other end of his perch before settling down again. Hardly knowing what to do, I started talking about my past and my relationship with a difficult mother. I still don’t know why I did that. He stopped me with a laugh. “Listen,” he said, “I had the world’s worst mother.”

I slowly began to realize he was looking kinder and kinder. Somehow I had turned the conversation in the right direction. As I left he walked me to the door, still keeping his distance. But then he leaned forward and said, “The floodgates are open.”

Sondheim was a genius—everybody knew that. He was funny and gregarious, too. But he could also be curiously odd. “There was always something very endearing about him,” Laurents told me. “At the same time, he was very defensive about his sexuality. I mean, he introduced me to the playwright Burt Shevelove, and one day I said to him, ‘You know, there’s something about Burt,’ and he said, ‘You’re wrong.’ And I said, ‘What do you mean?’ and he said, ‘He’s not a homosexual.’ He was very sensitive on that issue.”

Sondheim with his mother, Etta Janet Leshin, in New York, 1932.

Everybody knew of Sondheim’s gallows humor where his mother was concerned, such as receiving a platter as a gift and saying, “Thanks for the plate but where’s my mother’s head?” Everyone knew that the trouble started when his father left a note and disappeared in the middle of the night, and she crawled, weeping, into her 10-year-old son’s bed.

To spite her husband, perhaps—he ran off with a lover—she anointed their son the new head of the household. In her deranged state, she thought her teenage son owed it to her to substitute for her faithless husband in every possible way. She would, he said, sit opposite him and make crudely provocative gestures, skirts lifted and legs parted. He was by turns frightened and emasculated and aroused, as he doubtless described it during his long psychotherapy. Sometime during his teens he was approached by an older man and received an invitation to the kind of safe alternative that homosexuality seemed to represent.

Sondheim was attracted to several women. There was the lithe and lovely Lee Remick, dancer and movie star, who was married at the time but wanted a divorce. Sondheim said, “We would meet secretly in Greenwich Village like lovers.”

Another failed relationship concerned Mary Rodgers. She had known him since childhood, and after her divorce from Julian Beaty, Sondheim invited her and her three small children to move into his house in Turtle Bay. They were unofficially engaged. Mary, who was famously frank, described what happened when they tried to sleep together—in other words, nothing: “He was ice-cold with fear, and so was I. We were so terrified we seemed to be in a meat locker.”

By the time he was a Broadway hit, Sondheim had a reputation for kinky sex that involved paraphernalia and dominant-submissive games of one sort or another. I had reassured my subject that I was interested in his psychic energy, not his sexual one, which was true, and the only book editor who took me to task was a lady in a waistcoat working for a newspaper in San Francisco. His sex life was his business.

The only time it mattered had to do with his heart attack, when I should have included it in my book and didn’t. The story had to do with “poppers,” or amyl nitrite. They were popular in certain circles because when inhaled during sex, they relaxed anal muscles, but they were also capable of imposing a dangerous strain on the heart.

Sondheim had used them before. This time was different. One night in 1979, shortly after Sweeney Todd opened and during a sexual encounter, Sondheim began to sweat heavily and his heart started to pound. Suddenly he felt the worst pain he had ever known. “I just wanted it to stop,” he said. It was a heart attack.

I made various assumptions when I was writing this book. One of them was to guess that, as he matured, Sondheim and his mother grew apart. This never happened. Much as she had supported her husband’s career, “Foxy” Sondheim, as she was nicknamed, transferred her role of helpmate to her son.

She was the one who bought a farmhouse, in Bucks County, close to the Oscar Hammerstein family so that her son could be casually introduced. That was a smash hit. Her job was to steer the right people in Sondheim’s direction on every social occasion. She sang his praises everywhere she went and surrounded her guests with needlepoint cushions designed to celebrate each new show. She popped up at opening nights and stood at press conferences, smiling in the background.

He supported her every year, and it cost him $80,000, a lot of money in those days. When I asked him why, he mumbled something about feeling obligated and being “a good Jewish son.”

In the days before the heart attack, people remember, Sondheim was smoking and drinking, using cocaine, and losing interest in his appearance. He might wear the same clothes day after day and forget to shower. “He tended to nif’ a bit,” one of his friends said. He never combed his hair, and his mother would growl, “What did you comb your hair with? A rake?”

After the heart attack, his attitude changed with a vengeance. He stopped smoking and sniffing cocaine. He exercised, went on a diet, and lost weight. By the time we met, he was boasting about hating fish until he had to eat it, and becoming a convert in a week’s time. He installed a stationary bike and would watch old movies while he pedaled away. He never ate lunch anymore. He lived to be 91.

Another of my assumptions was that his musical-theater subjects were random, having nothing to do with his life. I was wrong about that too. I have come to believe they had everything to do with his life. I should have looked harder at those framed puzzles hanging on the walls, those highly stylized clues, each one concealing a significance to its owner’s subtle mind. Put together, did they have some coherence, spelling out in plain language the truth he could not reveal to anyone? After all, he loved the idea of “hidden in plain sight,” behind which he could hide, unknown, unknowable. What truth was hidden? It spilled out everywhere. It was revenge.

Take his first solo success, Company. Sondheim is clearly Bobby, barely disguised, having a surprise birthday party. Here they come, two by two, couples with big smiles and secretly failing marriages, all urging him to get married. He sings, “I’m ready!” But he is not. How can he be, when he is skewering his mother, someone narcissistic, materialistic, demanding, and empty-headed, as in “The Ladies Who Lunch”? “Everybody rise!” he repeatedly urges his audience. They don’t. They can’t. It’s too sad.

Sondheim at the piano, at home in New York, 2009.

Was he the ugly girl in Passion, the one who dares to love the other, the brute doomed to wander through his haunted castle until awakened with a forgiving kiss? What about Sweeney Todd and the barber, wrongly accused and sentenced by an evil judge, who, once freed from a hellish prison, sets out to murder his tormentor? Whose curious barber’s chair has a way of condemning its customers to becoming meat pies? Has revenge for his fate led him to an even more bitter discovery, that he is a murderer at heart?

“There it is!” says Judy, wife of Sondheim’s famous collaborator, Hal Prince. “The story of your life!”

Perhaps the most poignant song Sondheim ever wrote about himself is the finale number of Company. It is called “Being Alive.” When I first heard it, and in the context of the musical’s apparent theme, I made the natural assumption that it concerned marriage.

I believe now that that was a mistake. I believe the composer was referring to maternal love, selfless, all-encompassing, and out of reach. He took an enormous risk, even thinking about it. To admit that such a need was hidden behind the mask of ridicule and contempt—to admit his pain—was to abandon a lifetime of defenses.

A young stage manager who caught a glimpse of him while he was writing “Being Alive” was shocked. He said, “He looked terrible. This was the first time I realized the agony Steve was going through. It was painful and horrifying.”

What I missed—or, rather, never saw—was what it took for Sondheim to cry out for all that he had missed. “Someone to crowd you with love,” Bobby sings. Did he ever find it?

Meryle Secrest is a Washington, D.C.–based, Pulitzer Prize– and National Book Award–nominated author of several biographies, including of Frank Lloyd Wright, Stephen Sondheim, and Elsa Schiaparelli