R’ Tzadok haKohen has a tremendous answer to this question, but since R’ Reisman discussed that in his parsha shiur that people get via e-mail, I’ll give you something else. In case you haven’t seen it, the R’ Tzadok is in Takanas haShavin on 52b and, in a nutshell, he explains that the malchus went to Yehudah davka because he had failed. It’s the ability to climb back from the depths that is a mark of true leadership.A few weeks ago we discussed how Yosef’s punishment for relying on the Sar haMashkim was fitting davka for someone like himself. Yosef showed such great bitachon in trusting that his dreams would come true, to the point of making no attempt to contact Ya’akov all his years in Egypt lest Ya’akov hinder his efforts to bring those dreams to fulfillment. Therefore, Yosef should have shown the same high level of faith and not relied on the Sar haMashkim. You can’t have different standards in different areas of life.
Here we see the same idea, as Ya’akov hoists Reuvain by his own petard. Chazal tell us that Yehudah had no rest after death; he was barred from olam ha'ba. Moshe Rabeinu davened that Hashem should let him enter in the zechus of his having inspired Reuvain to say viduy. When Reuvain heard Yehudah admit wrongdoing and take responsibility for acting wrongly toward Tamar, he confessed his own guilt as well. (Parenthetically, as R’ Leib Chasman points out, we see that inspiring others is a much greater zechus than doing good oneself. It was only the fact that Yehudah inspired Reuvain that earned him entrance to olam ha’ba; the fact that he personally confessed was not enough.)Reuvain was impetuous, headstrong, first to act. He immediately took action and moved Ya’akov’s bed out of Bilhah’s tent and into his mother’s after Rachel’s death. So where was that zealousness when it came to saying viduy? True, Chazal tell us that Reuvain was the first to do teshuvah, but why did it take Yehudah’s saying viduy to inspire him to confess his own wrongdoing? Why was he not first to act there, when it came to admitting his own faults, not correcting other’s?
That’s why, explains the Ohr haChaim, Reuvain lost the malchus, and Yehudah, who took the lead in admitting his own faults and thereby inspired Reuvain, took his place.