For most of its history, Ponza was a penal colony. Now the island is a favorite vacation spot among the Romans. After an hour-long train ride to Anzio, they surge onto the ferry that shears through the Tyrrhenian Sea, which is bleached of color and flat-ironed by the midday heat. Ponza’s houses, rising behind the curved rim of the 18th-century Bourbon-era port, pile up like scoops of gelato in shades of melon, strawberry, and pistachio. Noisy three-wheelers—mostly the Piaggio Apé—wait to transfer luggage to simple guesthouses such as Villa Laetitia, a six-room bed and breakfast on a promontory, or Hotel Gennarino a Mare, a wonderful restaurant-with-rooms on a sandy beach.

In Ponza, everyone dances to the islanders’ tune, and the international jet set is not given much attention. The lack of five-star resorts and accessible beaches deter many potential visitors. As a result, Ponza remains gloriously uncontaminated by mass foreign tourism. “Come back at the end of September, when the swallows are en route to Africa,” says Enzo, the proprietor of the Da Enzo trattoria. “Even the Romans have gone home. That’s when it’s magical.” There are just two four-star hotels on Ponza. Grand Hotel Santa Domitilla is an appealing option, largely for its celebrated restaurant, Il Melograno. Most of its rooms lack sea views, so if such an amenity is non-negotiable, contact Bellini Travel, an agency specializing in Italy, to rent Casa Fontana.

Mornings just look better at Casa Fontana, where the sunrise views are postcard-perfect.

There are four lovely bedrooms, and the scented gardens are balanced on a clifftop beneath an umbrella-pine tree. Diamante D’Alessio, the proprietor, is a stylish former magazine editor, and she will arrange a dinner that showcases Ponza’s bounty—fresh figs, oysters, a fat snapper, pressed fish roe, bottarga with pasta, and custard-filled flaky pastries from Gildo, a local bakery.

After lunch, it’s not a bad idea to follow the crowds to the Frontone Beach Club, accessed by frequent boat shuttles that run all day from Ponza’s main port. Arriving at the club, turn left and head toward the quieter section of the beach, called Cocoa. The sun loungers and parasols are less expensive, and the charming little clapboard café—on stilts—serves a mean espresso.

The most stylish way to arrive at Frontone Beach Club? By one of the boats that shuttle visitors from Ponza’s main port.

The headland is crowned with ruins of an ancient castle, which is accessible from Cocoa Beach by a short swim. It was built to defend Ponza against the pirate raids that continuously devastated and depopulated the island throughout the centuries, leaving it best known as a place of exile.

In Ponza, everyone dances to the islanders’ tune, and the international jet set is not given much attention.

The Roman emperor Augustus incarcerated his adulterous sister Giulia here, and during World War II, the confinati a Ponza (confined to Ponza) prisoners were simply those who dared to oppose the Fascist regime. Then the tables turned and, in 1943, Mussolini himself was imprisoned at the butterscotch-tinted Pensione Silvia, which is now a hotel.

Ponza’s cheerfully painted houses loom large behind the 18th-century Bourbon-era port.

During the daytime hours, explore Ponza’s other seductive beaches by gozzo, the traditional wooden fishing boats that can be hired at any port. (No need to reserve in advance, except in August.) Ideal for navigating through hidden coves and caves, they will also drop anchor in empty bays to allow for swimming.

Given that eating well is Italy’s national pastime, it’s advisable to dine at the Michelin-starred Acqua Pazza in the center of town. At Da Enzo, a small trattoria hidden in a cave-like structure on Frontone Beach, the outdoor kitchen is spotted behind curtains of bamboo. Da Enzo is easiest to access by boat; it offers a free transfer from one’s ship or Ponza’s main port, and Enzo himself may be your captain. The simple, set menu is chalked onto the blackboard. The setting is magical, but it is not exactly undiscovered: Madonna and her retinue, traveling by yacht, had visited the week before.

The sunset views from the Chiaia di Luna hotel’s Le Terrazze bar go down best with an ice-cold glass of Etna Bianco.

Whatever you do, don’t skip the evening cocktails. On Ponza, aperitivo is a cherished ritual, particularly at the harborside Bar Tripoli and at the Chiaia di Luna hotel’s Le Terrazze. The latter requires flagging down one of the island’s few taxis, and a five-minute ride will cost you $15, but no matter—the sunset views over the water justify the expense.

There will also be D.J.’s and “mixologists,” along with influencers taking selfies. In addition to the delicious cocktails, the hotel’s restaurant is excellent. Order pasta with prawns, accompanied by a glass of Fieno di Ponza, the local white wine, on the deck, and enjoy the view of the sandy beach below. Here, according to mythology, the sorceress Circe seduced the wanderer Ulysses, temporarily dispensing with his shipmates by turning them into swine.

It was on Ponza where Circe was said to have seduced Ulysses and turned his shipmates into swine.

After dinner, it is only a bit slower to return to the port on foot, provided too many Negronis haven’t been consumed. But beware the streams of vintage scooters and toy-scaled Cinquecentos, which are available to rent.

The island’s main two-lane road is also occupied by public buses. For a $2 fare, they provide an adrenaline-fueled ride on switchbacks above white sea cliffs of volcanic rocks that periodically crumble like amaretti into the blue-green sea.

Casa Fontana is available to rent through Bellini Travel, with rates beginning at $12,000 per week

Catherine Fairweather is a West Country of England–based writer