Are you talking about something like uncanny valley? (animation that looks almost human but not quite enough to be convincing and causes a sense of unease in the viewer) Because if you are I find animation styles like Strange Magic(UUUGH)s and Mars Needs Moms' to be really really unsettling to look at.
I don't know about cartoons per-say but I HATE drawing gore or seeing very gruesome gore (Entrails) in cartoons/anime, it's just absolutely disturbing X_X
Also the fear you're talking about is Kyrofelonoshophobia, its more common in children then others due to a cartoons odd look and a child's ability to make weird things terrifying.
So when I saw Finding Dory in theaters, the uncanny valley and their shiny eyes sort of creeped me out. Everyone kept saying little Dory was cute but I thought she was the creepiest of all.
Also the reboot to Monster High14
I loved the original Monster High but this reboot...it just looks really creepy maybe because they look so dead behind the eyes.
@NickRowler
They want my soul, I know it!!
--runs away screaming--
I'm not afraid of any animaton style, but there are a few that I find a bit unsettling.
Uncanny Valley has been mentioned, and it can be very unsettling. It's worse in real life - with those creepy humanoid robots people are building (footage of those give me goosebumps) - but I find it difficult to be entirely comfortable watching 3D-animated humans that are very nearly realistic. Like, the Final Fantasy movies, or the Space Captain Harlock-movie. I spend a lot of the time with a small part of me feeling like something is wrong. Their eyes don't move quite right, their facial expressions change a bit too slowly, etc. - but on a still frame, they look human. It's weird.
I also am not fond of the animation style of things like Courage the Cowardly Dog, though that might be more of a holdover from my childhood dislike of the show itself - it, and several shows that were visually similar to it, were kind of loud and shrieky, and I was (and still am) fairly sound-sensitive when it comes to screeching noises, so I might be associating the two.
And as @NickRowler mentioned - the dead-eyed look is creepy! I find it's worse in 3D than in 2D, though.
Hahaha oh my gosh yes. The first time I watched Princess Mononoke and that boar burst out of the woods, I nearly choked on my tea. >.<
generic anime/manga, not real good anime/manga, just generic ones produced by random people who want to be the big mangakakaka they look up to. I see it all over comic sites and it's painful. I rarely ever follow manga comics because of this... most seem really uninspired and lazy.
Bad 3D. It's everywhere, so much so people think amateur 3D artists are just the norm. There's this trend with horrifying CG bimbos I see everywhere. Their lips look like gummy candy, clothes clipping into their flesh and their bodies made of Plasticine.
Characters with no noses and large mouths that take up the face. I've been seeing some SU/anime inspired artists take this approach and it is... horrifying. 12
While I love Disney films like The Little Mermaid and Beauty And The Beast etc. the original Disney crew creep me out. Mickey Mouse, Minnie, Pluto, Donald Duck, Goofy, all of them. Creepy. I could never go to Disney Land.

For some reason the cinematics in the old game Dark Earth scared the CRAP out of me when I was young and still kind of creep me out -
(Beware: Violence!)
Though it wasn't even the violence that was scary to me, it was their FACES. Just....WHAT THE HECK IS WITH THEIR FACES?! Their overall movements are pretty disturbing, too.
Man, I had a lot of those as a kid. And yet somehow I still kept watching them! I know everyone admires Ren & Stimpy for its animation/inventiveness, and as much as I respect it, I could never watch the damn thing. It was just too gross. I also had a thing about CatDog, there was this one ep about teeth that just drove me up the wall. Though, even cartoons I did like had a super-disturbing episode here and there.
As a kid I watched a lot of Russian cartoons and pretty much anything involving claymation/puppets was scaring me. They were just so....ugly. I just couldn't It's not until Chicken Run that I found out claymation doesn't have to be ugly and weird as hell.
there's a film called something like The Fantastic Planet with a really weird animation and art style, about big blue aliens. I physically cannot look at it without feeling incredibly nauseous... aside from that, I have a phobia of this like...particular kind of 3D animated floating orb kinda deal, it terrifies me.
Other than that i get easily squicked out by american gross-out cartoons like CN's golden age shows, those painted detailed close ups (a la spongebob) and any kind of art involving brains or eyestalks.
i think once your able to get over it's abstract art style it's a pretty enjoyable watch. I remember watching it on vhs with family as a little kid and even though the monsters were kind of creepy I thought it was pretty cool overall. If anything I think it's odd mixing of style was it's greatest selling point, aside from it featuring music by the Beatles, and was probably perfect having originally come out in i think the late 60s or 70s?
I think that like others have mentioned uncanny valley is the most unsettling. Then there's stuff like G-man from half life, and I've never once played the game but I've seen the one clip everyone seems to quote "Wake up mr. freeman"5 and anything that resembles that gives me the spooks. There's also Chulip, a video game yes, but between the graphics and probably the atmosphere of the game make it unnerving even if it has a "cute" appearance.