When we think of John F. Kennedy Jr., we mostly think of him in the early years of his life, or at the very end. We think of him as a child hiding under his father’s enormous presidential desk in the Oval Office, or saluting his dad’s coffin after his assassination. We think of him riding his bike to the Manhattan offices of George, the political magazine he founded, or holding hands with his glamorous wife, Carolyn Bessette.

But as I spoke to many members of the Kennedy family as well as to dozens of their close friends and neighbors for my book, White House by the Sea: A Century of the Kennedys at Hyannis Port, another picture of John emerged. Outside of Washington and Manhattan, John had a rich life on Cape Cod that had never been explored. The Cape was his safe haven, a place where he could avoid, for the most part, at least, the ever present lens of the paparazzi.

John F. Kennedy and John junior in Hyannis Port on Labor Day of 1963, the last summer they spent together before the president was assassinated.

For generations, the Kennedys have been protected in Hyannis Port, a fiercely private little community on the south side of Cape Cod. John’s family bought their simple, two-story clapboard house there in 1957—three years before he was born—for $45,948. It sat just behind the “Big House,” a beautiful waterfront home that his grandparents Rose and Joe Kennedy purchased in the late 1920s, and a third house, where his uncle Robert F. Kennedy lived with his wife, Ethel, and their loud, rambunctious children.

The Cape was a place where John F. Kennedy Jr. could avoid the ever present lens of the paparazzi.

After J.F.K. died, John’s grandfather Joe bought a reconditioned World War II two-seater observation plane as a gift for all of his grandchildren to play in. But John, who was five when they got the plane, loved it the most. John usually avoided the football games on the lawn, preferring to tinker in the old green plane.

John and the other Kennedy kids celebrate their grandfather Joseph P. Kennedy’s birthday, in 1965.

John and his cousin Tim Shriver also liked to go diving, spearfishing, and sailing on Nantucket Sound. They waited for the windiest day to go out on their little flat-bottomed Sunfish sailboats and surf on the waves, the dinghy flipping into the wake and righting itself again, the boys laughing as they gasped for air before flipping under the next wave.

As John got older, his mother, Jackie, began spending more of her time on Martha’s Vineyard, where she built a sprawling home on 375 acres. Even after Jackie left, John brought his friends from college and then his girlfriends to the house in Hyannis Port.

Bessette, whom he married in 1996, loved the charm and history of that old, weathered house. The first time she visited, she told John’s childhood best friend, Billy Noonan, “I like coming to this house. I went down to Palm Beach with John, and the place was creepy. There were too many ghosts down there.... This house has got a nice vibe to it.”

John walks on the beach in Hyannis Port the day after his grandmother Rose Kennedy’s death, in January 1995.

As John and Carolyn were more and more hounded by the press in Manhattan, Hyannis Port became their escape. They hosted intimate dinner parties for their friends, like the ones John’s mother used to have. And they slowly began to renovate the home, though John wanted the house to continue looking like it always had.

John, Carolyn, and her sister Lauren were on their way to the Cape on July 16, 1999, when their plane went down over the Atlantic. They were headed for the wedding of John’s cousin, and, while in Hyannis Port, John and Carolyn had planned to meet with the interior designer who’d worked closely with John’s mother and grandmother on their Cape homes.

They never got the chance to finish their renovation, but the tight-knit Hyannis Port community still shares stories of their memories of the president’s son they watched grow up there.

Kate Storey’s White House by the Sea: A Century of the Kennedys at Hyannis Port will be published on June 27 from Scribner