Another potential reason people like them is it gives them a few more goals to achieve other then "beat the game" and can make the game last longer.
This is THE reason for achievements. Not only does it give you a clearer sense of things you might want to try to do (they're almost hints sometimes, or clues to what's available to unlock), but it gives you ideas for things to try you might not have thought of. Also, when it comes to thinks like "find every ___ in the game", it may be a goal you thought of but not one you were willing to keep track of yourself. When the system can tell you if you've done it or not (at least in games where you don't know how many of whatever event are available), you're more likely to go through with finishing that goal. It's the same thing with points, there are games I wouldn't have kept track of points in that I now will because I know what numbers to aim for and when I've reached them. I hate when you've essentially beaten the main part of the game and then still keep trying to play it, because it's just aimless and useless. But, achievements give you more things to complete, and make playing games you've otherwise "finished" more interesting. Hell, they were about the only thing that made Beautiful Katamari worth playing a little longer, because you can only COMPLETE A STAGE about twice before you have no reason to ever again. But, if it gives you a different way of looking at that stage, it can be fun again to do things differently.
Sort of like Sarevok said though, I don't get winning achievements for basic parts of the game. Finishing something you have to do isn't worth having an achievement for, but doing something you might otherwise not is more legitimate. It convinces you to play the game a little more instead of rushing through it.
I don't know anyone that really gives a shit about gamerscores though, it's more interesting to say "I got all the achievements in this game" than "My gamerscore is this" because it could just mean you got all the easiest achievements in a shitload of games.