Most of those people are unreachable to the rest of the world - and even if they could, we're all so en cultured and socialized differently that we wouldn't get along anyways.
The main reason is simple: the people generally desired are taken, the ones undesired by those around them aren't.
Seriously, would you wanna date most women? Most are unattractive, dumb and piss you off with their retard conversations that could make you want to put an icepick in your brain. Vice versa goes for most men to women. It makes the selection difficult knowing that unless you yourself have some grand skills or qualities (looks, talent, cash, ideologies, etc.) that there is little reason for somebody to take interest in you - unless they're lonely/unattractive/poor/boring themselves and want somebody to relate to/can't get anybody else.
Happy Valentines Day!