Oooh! Ooh! Ooh! I've got some great answers to this.
I'll just start out with the names, and discuss each in detail. If you're in a hurry, just skip to 4 - it's the craziest.
1: The Talmud
2: House of Leaves
3: The Principia Mathematica
4: Gravity's Rainbow
1: The Talmud
The Talmud is an extremely important book, or books more accurately - though you may never have heard of it, I guarantee you are aware of its influence! It's a collection of teachings, arguments, sayings, and practice compiled over the course of about 700 years, dating from 200 BC to 500 AD, give or take. For the Jewish people, this is where a lot of laws are expounded about how to follow the holidays, keeping kosher, etc. If you know any piece of Jewish religious ideology, it is probably either in here, or in the Torah and expounded upon here. It also has a variety of bizarre fantastical tales of angels, rabbis shooting lasers out of their eyes, and more - but it's also very, very, very confusing and hard to follow. It's in Aramaic mostly, often incomplete sentences or acronyms used, though high-quality English translations with commentary to fill in the gaps exist (and surely for many other languages too), and a typical page is only about a quarter actual Talmud and the rest is commentaries accrued over the past 1500 years haphazardly arrayed around it in something that could be described as literally layering commentaries around a central column of text. The first time I saw a Talmud page, I literally couldn't figure out which word was the one I was supposed to start with! It's strongly discouraged to read it alone - pairs are typical - and an average reading speed is one page per day which completes the entire work in about eight years. I have read it almost daily with my study partners for a bit over a year now, and have read about 1/60 of the work. That's the kind of difficulty we're talking about.
2: House of Leaves
House of Leaves is... it's a book. It's a real... thing. I'll be honest and say I can't intimately remember what it's about - a man finds an extremely complex academic criticism on a film that doesn't exist, and reads through it, and the academic criticism gets bizarre. Its memorable quality, though, is the act of reading it. I can't spoil a whole lot without risk of ruining the experience, but just know that the layout of the page is important in a way that is not very common for most books. I understand Mark Danielewski's other books are similar in this regard. I highly recommend it, but know that it's among the scarier books I've read.
3: The Principia Mathematica
I haven't finished this one or even gotten especially far, though I do mean to. The cultural context of the book's existence is important to understand why it exists at all: It was created at a time when mathematics were in crisis and people felt it urgently necessary to create a system of axioms through which all of mathematics could be explained. The result is Whitehead and Russell's monumental work here, which tries to build up all of elementary mathematics through the simplest of building blocks - whether something is true or false. It's oddly humorous in places, like with immortal featherless bipeds in some examples, but the vast majority is extremely difficult, extremely dry applications of elementary logic that exhaustively builds up basic group and set principles until ultimately reaching 1 + 1 = 2 after about 800 pages - about the halfway point. It's so difficult that even the authors joked that nobody ever read the whole thing! Sadly, its goal ended up being proven impossible - no system of axioms does or could exist that satisfies what the book sought to do - but it nevertheless is an important piece of history that the few people crazy enough to want to read it (like me) will find interesting. I parody it in my novel, with the Principia Mathemagica.
4: Gravity's Rainbow
Oh boy. I don't even have the words to describe this one, but I'll try. Gravity's Rainbow was recommended to me in high school as a book to try if I "wanted a serious challenge". It took me a few years to get around to it, but I can confirm that the teacher was not joking, at all. I have never read a more difficult novel in my entire life, not even close, (The Talmud and the Principia are harder in their own way, but also are not novels) and I am reading Moby Dick right now. The plot stars a man named Tryone Slothrop who can predict missile launches before they happen through a means that is extremely NSFW because of experimentation on him as a child. He then goes on a bizarre quest to find a rocket that doesn't exist created of a mystery plastic, mixed with drugs and sex. Lots of drugs, and lots of sex. And I mean a lot of sex - if you found Game of Thrones too tame, then you'll feel at home. And musical numbers. And custard pies. And harpsichord toilet dimensons. And adenoid monsters. And musical numbers complete with stage directions and different vocal roles (soprano, bass, etc). And sentences that go on for entire page (of which there's about 800, by the way). There's dozens of subplots - dozens - and I have no idea what most of them even are. You'd be hard pressed during the first couple hundred pages to even figure out Tyrone is the main character, there's so much going on. There's a part where it just goes off talking about dodo hunting in the 1700s for several pages.
In order to read Gravity's Rainbow, I recommend you have some calculus-based statistics background, as well as at least an undergraduate level understanding of basic chemistry and physics, some double integrals, and ordinary differential equations. Knowledge of Jewish esotericism will be very helpful as well (Talmud, funnily enough, and Lurianic Kabbalah) - know some Tarot cards too. And no, none of this - not one word - is me joking or messing with you. It's all in there. All of it. It is absolutely maddening beauty, so much so that when it came out the Pulitzer Prize Committee was utterly split down the middle and ultimately did not give an award the year of its release. It breaks every rule of fiction and the result is one of, if not the, most difficult books of all time. I loved it, and when I finished it, I got up and walked away limping because I was so stunned. It took several minutes to start walking normally again.
You should read it. Yes, you.
