I'm not sure what the lesson is here. Graphics won't improve overnight and I don't think we'd ever get rid of all the things that can make a game annoying to a non-gamer, even if we'd want
to. Perhaps it serves as a reminder, then, that there remains an extremely high barrier to entry for games. All those things that are easy to ignore, second nature or even brilliant for us are completely alien to a non-gamer. Maybe some of us like it that way, seeing greater accessibility as a sign of a 'casual onslaught' that will surely destroy us all. Whether that is true or not, you shouldn't be surprised when your non-gamer friends and family look at what you're doing and wonder what on earth you see in it.
Let's not be so arrogant about our favourite pastime that we think that anyone who doesn't like it is just ignorant or doesn't understand. Perhaps they understand all-too-well and have simply come to the conclusion that it's not for them.