Well I mean...you could always just write the villain being a villain. People like seeing people do evil crap.

For 4, one of my favourite series ever is the League Of Super Evil. does a fantastic job on the really petty villainy like holding a neighbourhood barbeque without inviting the neighbours like the villains they are! But yeah, in prose little trickier.

I suppose the most obvious and reasonably well done version of the villain protagonist that could work in prose is Light from Death Note. I think what made him compelling was the schemes and in many ways his slow descent from well intentioned to just insane with power. Getting to see his intelligence and wits was really good and the slow decay was pretty good too. It's an example of a style I could really see working in prose if the writer was skilled enough. I think what made it good was that he started out well intentioned and in a place where you could get on board with him (how many heroes go on revenge quests after all) and then the cat and mouse game of will he get caught holds your tension, and then the ever growing sense that he's just mad and everything is spiralling out of control.

The other option of course is the Agatha Christie method of writing from the pov of the murder and just not telling the reader until the end, but then did you really write a villain protagonist or did you write a twist?

And I guess depending on your pov, heist movies could be considered a villainous, or at least anti-hero protagonist story. And I think similar to Light, a lot of what people enjoy are the schemes and also the team. So I guess humanizing a villain by a team of villains and their relationship could help, plus impressive scheming that makes the reader want to see how they pull it off. Magnificent Bastards are a very popular character type, after all.

I'd say a lot of popular TV shows fall under 2. (Breaking Bad, House of Cards, etc) I don't tend to enjoy that sort of show, and a lot of people try to justify the characters actions, but I think a lot of people just enjoy the human drama of it, or even enjoy rooting against the protagonist, kind of a morbid fascination with how far they'll go and hope somebody stops them.

Other things I've seen
-You don't initially realize the protagonist is a villain, and the story gets you on their side, only to play on the slow horror of what they actually are. (The movie "He Loves Me, He Loves Me Not" comes to mind)
-The protagonist is framed as a semi-sympathic 'lesser evil', where the audience roots for their victory against worse evils while hoping they'll avoid their worse natures ("Dexter")
-Psychological examination where the point is to explore how a terrible person thinks or became the way they are without gearing towards redemption.
-A tragedy framed around a semi-sympathetic villain who is destroyed by their own flaws (Macbeth? Do people sympathize with Macbeth)
-Something that plays on petty vengefulness and the desire to watch unlikable characters be hated.

I think the idea that the audience will choose who to 'root for' underestimates the desire to sit back and watch a train crash. (For self defense,) True crime is huge, and tends to focus on the killers rather than the victims. Many horror movies make no effort to make their leads likable, especially slashers.

So as someone who is writing a villainous type protagonist for my comic, I'll try and share my perspective. (Not that I'm super qualified or anything! )

So you mention these 4 ways that you think a villain could be the main character, and number 2 really stood out to me because it was the only one that didn't rely on comedy or misunderstanding. But it's also the "weakest" one you mentioned when it really doesn't have to be.

Here's the issue:

Why does the villain have to justify and explain constantly?

Does it make sense for the character or is it just to prove something to the reader?
Because if you want to prove to the reader why your protagonist is doing evil things, it's much more compelling to show those things directly to them. It doesn't have to be perfect reasoning, just understandable. And if a character thinks they are doing the right thing, show that through actions instead of just repeating dialogue. A character with a stronger sense of righteousness will actually spend less time explaining because to them there is nothing to explain. Also, it's best to avoid too much angsty dialogue at once lol.

A good character just needs a good motivation. If you can just give your protagonist that, it doesn't matter if they are good or evil. Then you just have to show the struggles to fulfill that motivation (aka conflict).

Most of the best villains are compelling. They have a sense of charisma even if they aren't people-oriented. They have to have at least some attractive qualities for the reader to want to continue the story. No person is purely black or white morally. Just like a hero has flaws, a villain will have virtues.

And how do you end their story? Well you can have them fail in a justified way, or maybe they succeed and you see how the world is affected by that.
Even if the protagonist is evil, most readers will kind of root for them anyway since that's what stories train us to do. So don't be afraid to think that's an option!

Overall, writing a villain is a lot harder than a hero protagonist but it's not impossible. Other people in this thread gave great examples that are worth checking out! And not everyone will enjoy the character, but that's true of every character and story.

Hope you get some answers that help! I think writing villains is really fun so it's worth the effort.

Would this be another example? The protagonist is ignorant of the harm they do. E.g., an alien who torments experimental subjects but doesn't realize they experience pain because it's own species doesn't.

Option 2 actually describes Paradise Lost and Lolita very well, neither of which are amateur works. I also note that I, personally, quite enjoyed both. The issue with this sort of story, however, is that it makes it very easy to attract an audience that is actually sympathetic to, and agrees with, the protagonist.

A lot of people are actually surprised when they learn that the message behind Paradise Lost is not "Satan was right, actually" but that, "Satan will mislead you with very attractive lies but in the end he's just a miserable loser who wants to drag everyone down with him." In fact, people will read through the entire poem and not realize that message.

Same for Lolita - Humbert Humbert is a straight up narcissistic, abusive, and manipulative pedophile who completely ruins Dolores' life, but he's good looking, intelligent, and writes beautifully, so a lot of people come away from it finding him to be a sympathetic romantic. There are a lot of contemporary reviews of Lolita that even describe the story as a tragic romance, even.

Grendel by John Gardner is somewhat a combination of 1 and 2 - he is still very much a monster who eats people, but the story is told from his perspective and it's hard to argue what his morality should be considering his nature. It's not so much that he's misunderstood, and he never finds acceptance, but given everything you learn about him and his life, you're pretty much left with, "Well, what else is he supposed to do?"

I sort of like #2 the most. Like how you know someone is bad but they almost try to convince you that what they are doing is OK.
I feel like the Oncler from the 70s cartoon is that. And it sort of annoys me that they "made him nice' in the 3d film.
This scene is so great

Honestly a lot of the Dr Seuss villains are great. From the Grinch to Butter Battle protag who wants to blow up humanity.

I think another reason why some people may prefer villians is that they're in some aspect relatable hence their actions may be justifiable to them. I'm actually writing one right now on the side of a psycho serial killer, and including some to little relatable things made it easier (ie extreme ethics, morals). Like they don't have to be relatable 100%, but something they face may make the reader feel that they're real in a way with what choices they make, how they react to certain things, boundaries etc

Hoping you find what you'd like, as it's a fun way to push boundaries in writing!!

5-Yes, the protagonist is a criminal.......but he is charming and entertaining in a way.....

-You can make the villains do crimes that are not brutal, but are bad actions like stealing and conning. For example, Lupin the third, focuses in a band of thieves and their adventures/misadventures.

-they can still be bad people, but maybe there are some lines they are unwilling to cross, which helps humanizing them.

-Another way to make the villain protagonist and their team likable is to make them extremely loyal to each other, being like family and true friends. you mess with one, you mess with all of them.

-style: popular antagonists are cool in a way, maybe they are imposing, intimidating or they are cunning and charming. One way or another, they manage to fascinate the reader. This not only works for antagonists, but for evil protagonists too.

I think it's cool when the protagonist is an actual scumbag, makes for some neat conflict at times

I'm not a professional writer so I might be completely wrong about this, but in my opinion the whole "your MC needs to be likeable/relatable" is a lie. What you need them to be is compelling. As long as the audience finds them fascinating, they'll keep reading, even if the protagonists is a complete monster (at least to a point, some people will drop the story if they find the MC crossed too many lines).
For compelling villains I like to point out to the She Ra reboot. The villains aren't the main characters but they recieve plenty of screentime and you get to know them. They're enjoyable, even when you know they are the bad guys.

It's not so much a lie as a misunderstanding of what people are trying to say. By your own example, they're fascinating, which makes them likeable. When people say "make your main character relatable" they don't mean "make them just like normal people" they mean "give them an element people can relate to". It's the reason villains who care about 1 person, often a mother or single friend, are so common. Because almost all people can relate you "yeah I love my mother" or "yeah I love my bestie" too no matter how outlandish the rest of the story and character and that forms the connection. To go back to my Light example up top, one of his early goals was the very relatable revenge for the death of his mother, and when it came to killing his father, he couldn't, we were introduced to him as a bored high schooler. No matter how insane the rest of the story go, we'd formed those connections to those elements of him.

I don’t see why you couldn’t write the villain as your protagonist. That could easily be a good story, without the redemption arc or comedy, etc. Just write them as themselves. Often, people doing evil think they’re justified (or they don’t care), so it could make for a satisfying story seeing why this person thinks their obviously wrong moves are okay. Or seeing them do bad things just because they want to. The protagonist doesn’t have to be the hero in the story, just the main character.