I myself prefer London or New York to Boston, I give "things to do" priority over aesthetics, by and large; I even like SoCal

That said, I think Boston is a "prettier" city than either one of them, and offers some fantastic walking, aside from anything else (although Tonty's list is good and will more than fill four days).
When I was there in July (in town, for once, rather than north for work) we spent an afternoon in the once rough southside neighborhood with a friend who has lived and worked there for years, and I was stunned at the quality and condition of the victorian architecture. Just walked around a bunch, then had dinner in a great restaurant (as often happens, up and coming areas are anchored by new and fun bars and restaurants).
Cambridge offers also some wonderful walking opportunities, better than the Boston side along the river in my view, and walking from MIT to Harvard along the river, then dropping in to Harvard Square, will make for at least another half day.
Mount Auburn cemetery is also impressive, if a little harder to get to without a car.
I'm also a big fan of new places: there is something comforting about spending a weekend in a place I know well, when I want to rest, chill and do things if I feel like it, but nothing beats the excitement of going somewhere new, our most memorable trips have unquestionably been those we have taken to different places, even if I might not ever really need to go back there.