Yeah that puzzle is crazy hard.  It's not necessary to proceed though.  If you talk to cultist in the lower right, he'll allow you to give up and move on.  If I recall you just miss out on a sword (that will eventually be replaced anyway).

-Ryethe