Once upon a time I asked (here) what troubled Rashi about the text that motivated him to explain the pasuk "V'ahavta l'reiacha kamocha" as being a "klal gadol baTorah", a great principle. I saw that R' Shteinman asks the same question in his Ayelet haShachar, but offers no answer.
Putting aside Rashi, the more basic question that begs asking is what Rabbi Akiva's point is. Hopefully we pay attention to the little details of halacha as well as the big principles, so what is gained by describing the mitzvah as a "klal gadol baTorah"? We've touched on this one as well (here). Chassidus (e.g. see Ch 32 of Tanya) connects the mitzvah of loving one's fellow Jew with the mitzvah of loving G-d -- where is G-d found in the world if not in the heart and soul of one's Jewish brother? We know that every Jewish soul has its roots in a letter of the Torah. Loving one's fellow Jew is thus not merely a matter of civility, but is a "klal...baTorah", a means of expressing one's connection and love for G-d and Torah as well.
Rav Shternbruch offers another answer that I thought was very powerful. There is a tendency to separate the mitzvah of ahavas yisrael from other religious obligations. This bifurcation leads to two errors. Firstly, there are those who think being a good Jew entails no more than being a good person-- it's all about tikun olam, being a nice person, etc. You can enjoy your tarfus so long as you lead a socially redeeming life. Rabbi Akiva taught that "V'ahavta l'rei'acha" has meaning only if it is a klal gadol baTorah, if it exists in the context of a commitment to other Torah obligations.
Secondly, and this is the flipside of Rav Shternbruch's point, it unfortunately doesn't take too much searching to find stories of those who think of themselves as bnei Torah who put on a poor demonstration of ahavas yisrael (and kav v'chomer non-Jews). Just as one cannot accept civility or ethics as stand-alone values divorced from other Torah precepts, one cannot be a medakdek in mitzvos without a committment to being medakdek in ahavas yisrael. Civility is not merely about social norma, but is a klal gadol baTorah, an integral component of one's religious persona.