I really like Shina's take, though honestly, some things that would obviously be sports aren't included in the Olympics (at least not to my knowledge).
I have a very backwards reasoning to why I think Poker is alright to be considered a sport. I ran Cross Country in high school and thus personally think racing is something that at the very core defines a sport, no matter the means (feet, cars, swimming). So since NASCAR is a sport by that definition and I think Poker requires much more skill than NASCAR (heh) so Poker should also be a sport. I just have a really broad definition of sport and it doesn't require any physical fitness component. Honestly, Golf and Baseball can be played heavy middle aged people past their prime so being physically fit isn't a must in sports.