I guess it depends how "cartoony" you mean. Does King of the Hill fall into that realm? Because I'd find it hard to watch a drama where the characters looked like South Park, but the plot was entirely serious. However, King of the Hill does a nice job of having more realistic storylines and characters although it's still certainly a comedy.
Absolutely. Animation holds a word of opportunity for story telling, and I think a lot of people would be down for a purely grown-up animated film. However, Hollywood has very little faith in animation that's not kid or family friendly and very little know-how on advertising such films, so it hasn't really been explored so much in the USA. There's a hilarious comic called The Goon that is currently looking for funding for a film adaption (and looks AMAZING in the test pieces:[link]) but they can't get anyone to back it because very few people with money think an R rated CGI film would be able to make its money back.
I see a lot of people have named some serious cartoons already, like Akira, Grave of the Fireflies, Ghost In The Shell etc. Although they are cartoons, they are done in a realistic style and it works for them.
But by "cartoony" style, do you mean cartoony like Invader Zim, My Little Pony or Adventure Time etc?