maybe this is only coming to mind because i've been playing a lot of them recently, but some Zelda games have had fairly sadistic dungeons.
in the 2D ones, especially, there were a lot of pits that made you fall down to lower levels. in games with the Lens of Truth, there were similar tricks, but where you'd fall through what appears to be a normal floor, and that sort of thing would prompt you in the future to nervously check all areas with the Lens before venturing forward. making the player anxious and obsessive-compulsive like that is pretty mean.
and in Ocarina of Time (and others i think?) there were these hands that come down from the ceiling and grabbed you, taking you back to the beginning of the dungeon. those were good because they offered just enough warning before coming down, via seeing a shadow over your character and hearing a subtle noise. they were fairly easy to dodge and kill after having had a few encounters, but i was definitely annoyed and angry at myself the first few times where i got nabbed. :P
another sadistic game mechanic that comes to mind is stuff that occurs outside the playable game itself-- for example, being forced to listen to Mr. Resetti's long rants in Animal Crossing games when you change the console clock or reset the game. hehe. and another example would be the game Eternal Darkness: Sanity's Requiem. in that game, the player character is going insane and you get to witness manifestations of that, such as seeing the walls dripping with blood. the really torturous thing that could happen in that game was that in certain conditions, it would make it appear like your save files all got deleted, when they actually still existed. it would cause you to freak out a bit, and it would kind of blur the lines between the player and the character, which is cool. the idea of using gaslighting within game design is one i'm intrigued by.
i hope these examples are the kind of thing you're looking for~