Before the battle against Og, Moshe Rabeinu was given an unprecedented pep talk by Hashem and was told not to fear Og. While the size and strength of Og certainly (as described later in Devarim) might have given Moshe cause for concern, the success of the campaign waged against the equally mighty Sichon begs the question of why Moshe thought he had more to fear this time around.
Abarbanel suggests that given the defeat of Sichon, Og's perception of the threat Bnei Yisrael posed was that much more real, hence he was apt to fight that much harder. In the battle with Sichon, the Torah writes, "vaye'esof.. ya'yezitzi," that Sichon first had to rally the people together and prime them for battle before heading out to war. However, when it was Og's turn, the Torah just says "vayeizei" -- the people were already primed for war (Netziv).
Rashi citing Midrash takes a different approach. It was not Og's physical might which Moshe feared, but rather it was his spiritual merit. Rashi identifies Og as the "palit," the refugee from the battle of the five kings against four who came to report to Avraham that Lot had been captured (Braishis 14:13). Since Og went out of his way for Avraham, he was deserving of reward for his effort. Many meforshim point out that Rashi in Braishis writes that Og had ulterior motives for his actions. Og hoped that Avraham would be killed in battle and he would then be able to marry Sarah. Isn't that a contradiction -- what kind of reward does pushing someone into war to die deserve?!
One could argue that indeed, Og did not deserve any reward. Moshe did not have a window into Og's inner thought and motivation and therefore mistakingly thought that Og had been sincere in his actions. G-d, who knew Og's true motivation in coming to Avraham, revealed to Moshe that in truth Og had no merit to rely on and there was nothing to fear. The Tzeidah laDerech acknowledges that this reading of Rashi is a "chochma gedolah," it's pretty sharp, but it ultimately it's not fulfilling. We see that Og must have had some special merit given that he lived for centuries, from the time of Avraham until Moshe (according to other views he actually survived the Mabul - see Nidah 61). Og's long life was the schar pesiyos, the reward for his footsteps in running to Avraham when Lot was taken. Even if his intentions were impure, Hashem still rewarded Og for the positive that came out of his deeds. With this background we can understand what the gemara (Brachos) means when it tells us that Moshe chopped at Og's ankle with an ax. Why the ankle? Because the foot represents these schar pesiyos Og had earned.
What is amazing is that the reward for those footsteps, no matter how evil the intent behind them, was enough to worry Moshe even given the tremendous zechuyos of his own and of Klal Yisrael which stood on the other side of the scale. How to understand that is beyond me. My only thought is that perhaps the lesson Moshe took away from Og's life is that G-d uses even the sinister intentions of evildoers to accomplish his plans. Whatever the selfish reason Og had for arousing Avraham to go to war, ultimately the result was the rescue of Lot and a kiddush Hashem. Perhaps Moshe began to question the need to wage war against Og, or even against any nation. True, these evildoers may intend harm against Klal Yisrael, but just as he did before to Og, can G-d not somehow turn around even the most evil plots and intentions so that good emerges from them? If bnei banav of the wicked can end of learning Torah in yeshiva somewhere, how can we justufy destroying them? G-d's message to Moshe was that these cheshbonos are his alone, but our mission is to eradicate evil when we see it.
The Zohar interestingly gives Og a bit more credit and writes that Og accepted the mitzvah of bris milah along with Avraham. Based on this, pshat in the following pasuk is a little different than we are used to:
וַיָּבֹא, הַפָּלִיט, וַיַּגֵּד, לְאַבְרָם הָעִבְרִי; וְהוּא שֹׁכֵן בְּאֵלֹנֵי מַמְרֵא הָאֱמֹרִי, אֲחִי אֶשְׁכֹּל וַאֲחִי עָנֵר, וְהֵם, בַּעֲלֵי בְרִית-אַבְרָם
It was not Avraham, but it was Og who was "shochein b'Alonei Mamrei," and it as Og as well who was one of the "ba'alei bris Avraham."
The upshot of this whole discussion is that there are strengths and weaknesses to both the pshat approach of the Abarbanel and the Midrashic approach of Rashi. According to Abarbanel we need not resort to extra-textual explanations of Moshe's fear, but it is harder to explain what made Og different than Sichon. According to Rashi, Og had merits that Sichon lacked, but it's hard to understand why those merits would cause Moshe such fear.