Stand on the lawn of Congress Hall in Cape May and the whole thing feels almost too pretty to have ever involved government work. There are rocking chairs on the porch, salt air coming off Beach Avenue, and the Atlantic sitting just across the street like it has been patiently waiting there since the 1800s.
But this little Shore town was not just a summer escape for families with beach tags and waffle cones. Five U.S. presidents spent time here, and Benjamin Harrison even made it his Summer White House.
That sounds like the kind of local-history brag you hear from someone who has had one too many scoops at Kohr Bros., but it is true. Cape May has always known how to make a beach trip feel dressed up.
Here, presidential history, Victorian porches, seafood dinners, and ocean breezes all seem to share the same address.
How Cape May became the Shore’s presidential escape
Before Cape May became the place where people rent bikes, book painted-lady inns, and debate the best breakfast spot with unnecessary intensity, it was one of America’s earliest seaside resorts. That matters.
In the 1800s, going “down the Shore” was not the casual flip-flop operation it is now. A summer trip to the coast was part health cure, part social event, and part status symbol.
Wealthy visitors came for ocean bathing, cooler air, and the kind of polished summer scene that made them feel far away from city heat without giving up comfort. Washington, D.C., in the 19th century was famously brutal in summer, with swampy heat, formal clothing, and no central air conditioning to rescue anyone in a wool suit.
Cape May offered a better option: sea breezes, grand hotels, social calls, wide porches, and enough refinement to make powerful people feel properly hosted. The town’s location helped, too.
Sitting at the southern tip of New Jersey, Cape May was reachable by steamboat and later by rail, which made it especially attractive to wealthy Philadelphians and political figures moving through the Mid-Atlantic social circuit. Presidents including Franklin Pierce, James Buchanan, Ulysses S.
Grant, Chester A. Arthur, and Benjamin Harrison all became part of Cape May’s story.
They were not coming for boardwalk chaos, neon rides, or late-night arcade noise. Cape May was quieter, more elegant, and just a little buttoned-up, even with sand underfoot.
That old identity never fully disappeared. The streets are still walkable, the porches still matter, and the town still carries itself like it knows exactly who has signed the guest book.
Congress Hall still wears its White House history well
The big yellow hotel facing the beach does not exactly blend in, and Cape May would be less interesting if it did. Congress Hall sits at 251 Beach Avenue, right where the oceanfront and the center of town seem to shake hands.
It is the kind of building people photograph before they even know why they should care. The hotel dates back to 1816, when Thomas H.
Hughes opened it as a boarding house. Locals reportedly teased it as “Tommy’s Folly” because it seemed too large and ambitious for the young resort town, which is a very Cape May detail: even the doubt had personality.
Hughes later served in Congress, and the hotel eventually took on the name Congress Hall. The original wooden structure did not survive forever.
Cape May’s Great Fire of 1878 destroyed a large section of town, and Congress Hall was rebuilt in brick in 1879, giving it the sturdy, stately look visitors still recognize today. Several presidents stayed there, but Benjamin Harrison gave the place its strongest claim to fame when he used it as his Summer White House.
What makes Congress Hall special now is that it does not feel like a roped-off relic. You can still see guests crossing the lawn in beach clothes, people settling into porch chairs, families drifting toward the Blue Pig Tavern, and wedding parties trying to look calm in the wind.
The history is right there, but it is not frozen behind glass. It is mixed into the regular rhythm of a Shore day, somewhere between the beach towels, the dinner reservations, and the rocking chairs that make everyone sit a little longer than planned.
The Victorian streets make the whole town feel frozen in time
Step just a few blocks away from Beach Avenue and Cape May starts showing off in a completely different way. The ocean noise softens, the sidewalks feel calmer, and suddenly the houses are doing most of the talking.
Turrets, gingerbread trim, wraparound porches, patterned shingles, stained glass, and bold paint colors appear one after another, like the town is politely competing with itself. Cape May’s Historic District is a National Historic Landmark, and that is not just decorative language.
The district includes more than 600 preserved Victorian-era buildings, which is a major reason Cape May feels so different from many other Jersey Shore towns. It did not save one pretty block and call it history.
It preserved an entire mood. Streets like Hughes, Columbia, Gurney, and Ocean are worth walking slowly, especially in the early evening when the light hits the porches and everyone seems to be either heading to dinner or pretending they are not staring at someone’s front door.
The Emlen Physick Estate on Washington Street adds another layer to the story, with its 1879 mansion and tours that help explain how Cape May’s wealthy summer crowd actually lived. But the best part is that the town does not feel like a movie set.
These buildings still work for a living. They are inns, private homes, restaurants, shops, and bed-and-breakfasts with porch furniture that actually gets used.
You might pass a historic plaque, then immediately smell coffee, sunscreen, or someone’s seafood dinner drifting through the air. That is Cape May’s trick.
It can feel frozen in time without feeling frozen in place. The past is everywhere, but nobody is asking you to whisper.
Benjamin Harrison left the biggest presidential footprint here
Harrison is the president Cape May gets to claim most loudly, and for good reason. Other presidents visited, but he made the town part of the job.
During his presidency, Congress Hall became his Summer White House, which meant Cape May was not just hosting a vacation. It was briefly hosting the business of the country.
In 1891, the actual White House was undergoing renovations that included the installation of electricity, and Harrison headed to Cape May with his wife, Caroline. The detail is almost too good: the White House was being modernized, and a little New Jersey beach town ended up with one of the best presidential history stories on the Shore.
Harrison’s Cape May connection also reached beyond Congress Hall. A group connected to Philadelphia businessman and Postmaster General John Wanamaker presented him with a cottage in Cape May Point, the quieter community just a short drive from central Cape May.
Cape May Point still feels like the right place for someone who wanted ocean air without too much fuss. It has dunes, cottages, birding trails, and the kind of hush that makes the busier blocks of Cape May feel farther away than they really are.
Harrison’s time here was both official and personal, which is what makes the story stick. Staff, correspondence, visitors, and national attention followed him, but so did the ordinary pleasures of a summer place: walks, breezes, porches, and distance from Washington’s heat.
Cape May does not plaster Harrison’s name across every surface, which is part of the charm. The clues are quieter: a historic hotel, a presidential marker, a Cape May Point detour, and the sense that this town once managed to be both a beach retreat and a temporary seat of power.
The beach town charm goes far beyond politics
The presidential history is a great hook, but nobody keeps coming back to Cape May just because Benjamin Harrison once packed a summer bag. The town has too much else going on.
It is small enough to feel manageable, polished enough to feel special, and beachy enough that you can still show up with sand in your car and no serious plan. The beach is the obvious starting point, running along Beach Avenue with the paved promenade just across the street.
This is not the wooden-boardwalk energy of rides, games, and giant prizes no one has room for in the car. Cape May’s promenade is calmer, better for morning coffee walks, evening strolls, and watching the sky turn pink over the water.
The Washington Street Mall gives the town its walkable heart, with pedestrian-only blocks filled with shops, restaurants, candy counters, bookstores, and the kind of benches that always seem occupied by someone holding a bag of fudge. Food is a big part of the appeal.
The Lobster House near the harbor is a local institution, especially if your idea of vacation involves seafood, boats, and no need to dress up too much. The Mad Batter has been a Cape May breakfast favorite for decades, and Congress Hall’s Blue Pig Tavern lets you eat in the middle of all that old-hotel history without making the meal feel like a museum tour.
A few minutes away, Cape May Point State Park adds another side of town entirely. The Cape May Lighthouse, built in 1859, stands 157 feet tall, with 199 steps to the top.
From up there, Cape May makes perfect sense: ocean, bay, marsh, rooftops, and a town that somehow made history look relaxed.
Why Cape May still feels grand without feeling overdone
Plenty of historic beach towns try so hard to impress you that you can feel the effort. Cape May does not have that problem.
Its grandness comes from what is already there: old hotels, preserved homes, oceanfront blocks, porch culture, garden paths, gaslamp-style streets, and a pace that slows down naturally after dinner. The town does not need to turn every corner into a spectacle because the setting is already doing plenty.
Scale has a lot to do with it. Cape May is compact enough that you can park once and spend hours moving between the beach, Congress Hall, the Washington Street Mall, and the Victorian side streets without feeling like you are managing a travel itinerary.
That makes the town feel elegant without feeling exhausting. Nothing has to be oversized because the details are strong.
A striped awning, a painted porch ceiling, a brick hotel glowing at sunset, a beach cruiser leaning against a fence — Cape May knows how to let small things do the work. The history helps, of course, but it is not the only thing carrying the place.
Cape May still functions as a real Shore town. People line up for ice cream. Beach tags get clipped to bags. Porch conversations stretch longer than expected.
Someone is always a little sunburned and insisting they are fine. That ordinary summer rhythm keeps the presidential past from becoming too precious.
You can visit for architecture and end up on the beach until sunset. You can come for seafood and accidentally learn about a Summer White House.
You can climb the lighthouse in the morning and sit on a hotel porch by evening. Cape May feels fit for presidents, but never reserved for them, which may be the most impressive thing about it.