Doylestown, the onetime home to Margaret Mead, Pearl S. Buck, Oscar Hammerstein, Stephen Sondheim, and assorted Mirvises, has 2.5 poured-concrete castles, 1 art deco cinema, and a surprising surfeit of museums. Wikipedia identifies the Bucks County seat as "a bucolic tourist destination" where "the gentry of Philadelphia and New York ... maintained country estates and often summered." M. Knight Shyamalon's modern classicSigns was shot just down the road.
Not to be outdone, the adjacent town of New Hope lies along the Delaware River and has counted (at various times) Andy Warhol, Aaron Burr, and Ween among its residents. The presumed subject of Low's 1994 LP I Could Live in Hope, New Hope has an "active and large gay community ... attracts motorcyclists (bikers) in the summer months," and "in January 2014 ... decided it would no longer hold weekly firework shows citing a rise in shoplifting."
Both towns are extremely quaint, filled with shops and restaurants, and located just 20 minutes apart from one another. New Hope and Lambertville are probably cooler. Doylestown is probably quieter (but has a really good record store).