I think you're right about being-read-as-characters, yes. It's something I struggle with when I write responses/reviews to memoirs, since I try, in general, not to make personal judgments of the authors... and yet with a memoir you sort of have to. I have come, finally, to the conclusion that if you write a memoir you're putting yourself in the public sphere, and therefore I don't need to feel guilty writing, "[Author] does a lot of stupid things and comes across as a bit of a twit."

And, along the same lines, while there's a certain extent to which a little self-absorption is defensible and probably necessary when you're writing the memoir, since it is about you... you don't want it to leak out onto the page. You-the-writer may need to be a bit self-absorbed; you-the-character-in-your-own-book had probably better not be.