Does age matter?

Now that it seems I am entering the world of the dating single people,(well, getting close anyway) the question of age has come to me. I have always had a thing about being with someone close to my age. I never wanted to date a much younger person or a much older person.
People make snide comments when an older man dates a much younger woman. Noone seems that comfortable with it. Now, I'm not sure what we say when a woman dates a man much younger. "good for her?"
My opinion is that people are at different places in their lives at different ages. I hope to be with someone that is somewhat at the same stage of life as me. But what if you meet a younger person and they seem to be at a good place in their life? Does that make it ok? Does someone have to have raised kids, been married, divorced, been to hell and back before they are worthy of dating?
It might just be refreshing to be with someone for awhile that's done none of those things. There would be far less baggage and their outlook on life would probably be a hell of a lot better.
I guess I'm asking myself does age really matter? Why am I so hung up on it?
I met someone the other night who was quite a bit younger than me. When he told me his age, my jaw dropped and I said "dude, I am 43!" and he looked at me all confused and said, "so?". I was a bit thrown, I was waiting for him to say Oh my god, and find a new seat to sit in. He did not do that.
Is 43 the new 32?
jb