In a shop selling lobster pâté on the Île d’Yeu—a 12-square-mile patch of beach and whitewashed houses off the French west coast—you can buy a can of local air. That’s how much its devotees love the place. But it remains somewhat of a secret. Even in France, few could locate Île d’Yeu on a map. (It is near the more well-known Île de Ré, off the Vendée.)

Port-Joinville, the island’s quaint port town.

Yet it attracts quite a set. The Belgian king and his family have spent summers here for more than two decades, recently buying a modest, tiled house amid pines. (By law, every house must be white.) King Philippe, Queen Mathilde, and their four children picnic and kite surf. Wall Street financiers and French actors—The Count of Monte Cristo star Pierre Niney, comedian Gad Elmaleh—mingle at the market in the hilltop village of Saint-Sauveur. Michel Barnier, briefly the French prime minister last year, is among the top politicians and journalists who go. However, “there’s no social hierarchy here,” a former mayor once said. “Nobody cares.”

Fanny Belhomme, who grew up in Lyon, visits the Île d’Yeu with her parents, three younger siblings, and the same family friends each August. “I’ve been going for maybe 20 years, and what I like is how every year nothing changes,” she says. “Your life moves on, but there’s something about the island that is—in French we say hors du temps, out of time.”

By law, the island’s houses must be painted white.

Hotelier Michel Delloye, who has been coming to the island since he was born, has called it “St. Tropez 50 years ago, Cap Ferret 20 years ago.” Unless you have a helicopter, the only way in is the 400-seat ferry from Fromentine, which takes about 30 minutes, depending on the sea. “When you arrive on the island, you put on your old espadrilles, you put on your old Bermuda [shorts], which have been the same for 20 years, a little stained, and you go to buy fish or go to the beach,” Delloye says. “You have a little glass of white wine, and nobody knows who anyone is.... People talk about what they are, not what they have.”

The restaurant at Les Hautes Mers.

Gradually, the secret is getting out. Last year, Delloye, known for his Parisian hotels, opened La Mission, a 22-room boutique hotel with a pool, spa, and quiet bar under the stars, in a Napoleonic garrison in Saint-Sauveur. It is one of two new hotels to open in the last three years, the other being Les Hautes Mers, on the edge of Port-Joinville, the island’s port town. Natasha Andrews, Niney’s Australian wife, gives yoga classes there.

Left, the pool at La Mission; right, the Pointe des Corbeaux Lighthouse.

The food scene is low-key: from fresh albacore ceviche at La Mission and farm-to-table dinners at La Ferme d’Émilie, to flaky strawberry tarts from the island’s Mousnier bakery. But there are few restaurants, and no parasols. The main attraction is the raw, unspoiled beaches: an incredible 30 of them.

On the Atlantic-facing Côte Sauvage, the beaches are tidal, orange-sanded, and backed by heathered cliffs. At the other end of the island you could almost be in the Seychelles. Fine white sand and dunes go on forever. There are stony bea​​ches, beaches for windsurfing and sailing. There is one with a ruined castle above it. There is a tiny one where Queen Mathilde likes to put down her towel in solitude. Plage des Soux is probably the most spectacular, a perfect U-shaped bay beneath a steep cliff.

The dunes of Grande Conche beach.

Cars are allowed on the island, but they are rare—some visitors hire open-top Citroën Méharis—so, to get to the sea, you cycle down dusty paths, passing faces you have seen several times that week. It wasn’t always like this. “In the 1950s, we had fishing factories … mainly tuna or sardines.... They’ve all gone now,” says town councillor Patrice Bernard, who was born on the island.

A bedroom at Les Hautes Mers.

Tourism took off in the 1970s. Moneyed Parisians went to Île de Ré, but the bourgeoisie of the provincial cities—particularly Lyon, Bordeaux, Lille, and Rennes—came here. Many of their children went to work on Wall Street, in the City of London, or in Asia, and when they returned they brought with them a cosmopolitan sensibility. Almost two-thirds of the island’s 6,000 houses are second homes. Every August, the population grows from 5,000 to more than 35,000.

The pool at Les Hautes Mers.

Each ferry departure is an event. Parents who are staying wander down the quay and wave good-bye to their adult children as they head out of the port, carrying with them their memories of a familiar paradise, and perhaps a little can of its air.

Richard Assheton is a Marseille-based writer