When Dallas Hayes was 11 he moved with his mother to an apartment building on Manhattan’s Upper East Side: a plain, square, redbrick place with narrow corridors, a small creaking lift and a fearsome woman from Alabama who lived in a little flat on the ground floor.

Hayes remembers being at a party on New Year’s Eve, in the flat next to hers, when this lady appeared in the doorway. “She said if we didn’t keep it down she would buy the building and have us all kicked out,” he recalled, standing in the building’s porch one recent evening.

He didn’t know it then, but his neighbor was Harper Lee, whose 1960 novel To Kill a Mockingbird became one of the most popular American novels of the 20th century and sold more than 40 million copies worldwide, even as its author managed a near-complete retreat from public life.

New York, the city where she came to make her name as a writer, had become the place where she successfully disappeared.

Now eight short stories found in her flat after her death in 2016 are to be published for the first time, offering a new window into the mind of a beloved writer and a glimpse of her early life in Manhattan.

“Four of the stories were known to have existed thanks to the archives of Harper Lee’s first literary agent, Maurice Crain, but were thought to have been lost,” said Michael Dean of the agency Andrew Nurnberg Associates, who negotiated the deal on behalf of Lee’s estate. “The other four are, to my knowledge, entirely new discoveries.”

While some are said to be set in the world of her childhood in Alabama and apparently show Lee working towards the themes and characters of her most celebrated novel, there are also stories about Manhattan.

“All the short stories in the collection date to the 1950s when Lee was an aspiring writer living in New York and yes, some are set there,” Dean said.

Asked if he thought Lee would want them published, Dean said: “Lee kept these pieces safe for decades. They are not rough notes or fragments; these are clean, typed, completed stories. Many of them have her address on the first page from when she sent them to magazines and literary agents for consideration.”

Charles Shields, author of the 2006 biography Mockingbird: A Portrait of Harper Lee, was thrilled by the prospect of new stories. “It’s exciting,” he said. “On the other hand, we may step back and learn to admire the To Kill a Mockingbird author for its craft and the writer’s ability in the light of short stories that aren’t quite as good.”

Lee and Truman Capote in Special Agent Alvin Dewey’s home in Kansas, where Lee was helping Capote with his research for In Cold Blood, 1960.

Lee, whose full name was Nelle Harper Lee, grew up in Monroeville, Alabama, where the boy next door was Truman Capote. In 1949, after dropping out of law school, she moved to New York, finding a flat in what was then a working-class German-American neighborhood on the Upper East Side, filled with German breweries, taverns, delicatessens and cinemas showing German films. It was “unlike anything she’d known in the south,” Shields said. “And that to me indicates a degree of adventure in her inner personality and a willingness to find out what’s beyond the limits of that little town, that postage-stamp-sized town in Alabama.”

She worked for a trade magazine and then as an airline ticket agent while trying to write in the evenings, tuning out the cacophony of noise outside. In 1956 two New York friends who had enjoyed a successful year gave her, as a Christmas present, the equivalent of a year’s salary so that she could write full time.

The following year she completed a novel called Go Set a Watchman, set in the 1950s and told by a young woman who is returning to her home in Alabama. An editor who read it persuaded her to recast the story, setting it 20 years earlier in the narrator’s childhood. This story went through several more drafts until it became To Kill a Mockingbird.

Lee on the set of To Kill a Mockingbird with Mary Badham, who played Scout Finch.

Four years after it was published, Lee told an interviewer that she had been expecting “a quick and merciful death” at the hands of her reviewers. Instead she won a Pulitzer, the novel became an Oscar-winning film starring Gregory Peck, and she was drenched in praise. She found this “just about as frightening” as the savaging she had anticipated, and struggled under the burden of it.

In the 1980s she began working on a book to be titled The Reverend, about a preacher in rural Alabama named Willie Maxwell whose two wives, a brother, a nephew and then a stepdaughter had all died in mysterious circumstances after taking out life insurance policies. Maxwell was then shot dead at his stepdaughter’s funeral by one of the grieving relatives. At a subsequent trial this man was acquitted, in spite of the hundreds who had witnessed the shooting.

Lee hoped to find, in this extraordinary tale, her answer to Capote’s non-fiction crime story In Cold Blood but it never came together. Tom Radney, an attorney at the center of the story, who had helped her with her research and apparently hoped that he too would soon be on the big screen, played by Gregory Peck, complained that Lee was “fighting a battle between the book and a bottle of Scotch, and the Scotch is winning.”

Lee divided her time between her home in Monroeville, where she played golf and occasionally answered the door to autograph-seekers, and her flat in New York. Her elder sister Alice, who lived to 103, became her attorney and gatekeeper, keeping the gate closed.

A few months after Shields began researching his biography of Lee, he received a letter from Alice. “What she said, I’m paraphrasing here, was: ‘I understand you’ve been talking to my sister’s friends and acquaintances about her life. I don’t like that,’” he said. But “it was not a cease and desist”.

He noticed that the letter came several months after he believes Lee learned about his project. He wondered if she knew, but did not tell Alice. “Perhaps there was a wish on Harper Lee’s part that this would be done, but without her compliance or her help,” he said.

In 2012, in declining health, not to mention the fact that she was 101, Alice retired and the lawyer who took over discovered the manuscript for Go Set a Watchman. Lee had believed that it was lost. She agreed for it to be published and it was in 2015, a year before her death.

Now the eight short stories found in her New York flat are to be published too in a collection called The Land of Sweet Forever, alongside some later essays written for magazines and a letter Lee sent to Oprah Winfrey. Edwin Conner, Lee’s nephew, said the family were delighted that the short stories had been discovered. “She was not just our beloved aunt but a great American writer, and we can never know too much about how she came to that pinnacle,” he said in a statement.

At the building in Manhattan where she was once the downstairs neighbor, there was some excitement too. Lee had previously lived at another address, across the road, but when that was demolished and replaced by a grander white-brick tower, she moved into 433 East 82nd Street. A woman who answered the door at the one-bedroom flat where Lee once lived said she had heard about the discovery of the short stories, but declined to give an interview, saying she had to take a work call.

Dallas Hayes, 56, a computer repairman, stepped into the porch moments later, carrying an electric scooter, dripping wet from the rain. When he moved into the building, in 1979, he did not see that much of Lee, or have the least idea who she was.

“She was just a scary old lady,” he said. Then he added, thoughtfully, that he was 11 at the time and at that age, all older women “are just all witches”.

“She was also cantankerous too,” he said. “She wasn’t like a ray of light”. He remembers her appearing in the doorway, at the New Year’s Eve party, threatening to buy the building. “I don’t know if she was joking,” he said.

After Hayes joined the US navy, his mother reported another encounter with Lee. They knew her by her first name, Nelle, he said. “Nelle says, in that southern drawl: ‘Where’s that boy of yours?’” His mother explained, and Lee replied: “Oh that’s great,” Hayes said, and he was a little tickled to discover that she knew who he was, even if she still remained something of a mystery to him.

Soon there will be a new book of her writing, and sometime after that, a new biography authorized by Lee’s estate, by the writer Casey Cep, and he and the rest of the world may know a little bit more.

Will Pavia is the New York correspondent for The Times of London