It's the latter; I don't know that anyone else notices particularly, and I'm actually well set up to handle it when other people are confused/bugged by me gender-wise. People read me as a boy/man *constantly* from elementary school through my 20s, even when I wasn't dressing masculine. Which freaked me out as a kid/teen, but I'm totally at peace with now, though it doesn't happen nearly as much now.