The Pardes Yosef answers by quoting a vort from R’ Yitzchok of Vorke. The gemara (Meg 28) writes that R’ Zeira said that he merited living a long life because (among other things) he never rejoiced in a friend’s downfall. It seems a very strange thing to boast about. Surely none of us would be happy to see a friend or colleague have trouble -- this doesn’t sound like a special midas chassidus that would warrant extra praise or extra reward. What was so special about R’ Zeira’s behavior?
R’ Yitzchok of Vorke explained that we are translating the words wrong. It goes without saying that R’ Zeira did not rejoice in his friends’ bad luck. None of us would do that. What made R’ Zeira special is that he never rejoiced – period – while a friend was in trouble. His sense of empathy was so great that so long as a friend was in pain, he was in pain.
Yosef may have held back only Shimon, but all the brothers felt as if they too still remained in prison so long as they were not together.