I'm curious to know why you feel that way. I always thought he was annoying at the beginning but I've come to like him.
His more irritating qualities didn't have to ruin him, but because of where the show went, they did. Again, none are Matt Smith's fault, rather they are the fault of the man who does not understand the character (or any character) and who cannot write a cohesive story to save his life.
He is a liar. About things that he need not lie about. In
The Girl Who Waited (which I actually enjoyed up until the moment where my heart gave up on the show) he lies to old Amy. My doctor would never have lied and sacrificed old Amy to save young ("
his") Amy. He would have figured out another way. It was disgusting. He's unfaithful to River, emotionally. I have no affection for River, but it is unacceptable, because it has been
shoved down my throat that
he has affection for her.
He is inconsiderate. He is selfish. Self-interested beyond belief. He may sulk and stomp his feet about how much he hates himself. He may wallow in his self-absorbed and immature shame, but none of his actions match the moments of angst.
He yells all.the.effing.time. It is not effective when a character does it all the time. It's not scary or a sign something is really wrong (like with previous incarnations) when he loses control, it's just what he does in every situation. Mystique gone.
Related to above, but really quite important, he is downright emotionally abusive to his companions now. Toys with Amy. His companions were never play-things before, but now I can't see them as anything else because that is the way The Doctor frames them.
Aside from a brief flash or two of things that happen to 9 & 10 - GONE. This makes him shallow. His character ostensibly forgot everything that happened to him before. When he regenerates, a new Doctor is expected - a Doctor who has highlights of the old person. He should be the antidote to rampant cynicism. He is damaged but good. What defines his goodness varies, but it should always shine through.
All of this (and more but I digress) creates an chasm which makes it impossible for me to connect to him, or his companions, in any way.