Bearing in mind that I can only speak for myself...
I kind of realized that after my intial attraction to someone, I immediately find myself wishing they were taller. : D I honestly don't know what the deal is with that...
Maybe you're having "the grass is always greener on the other side" thoughts.
What I have noticed, though, is that not very many tall women are willing to date shorter guys.
Prime example (laugh all you want): I had a huge crush on *swallows hard* Ryan Seacrest.
UNTIL I discovered Joel Mc Hale, who is basically a 6'3-6'4 Ryan Seacrest. And there's also the fact that since I was 13, I've only dated two guys under 6'.
In this case, the social privilege applies to short women.