One of the answers the Ohr HaChaim gives is that while in the short term things are about to get much better for Yosef, from the perspective of Jewish history, things are about to start going downhill. Yosef’s rise coincides with a famine that will cause his brothers to come down to Egypt (never mind the pain the famine itself must have caused) and cause his father Ya’akov to enter a galus that will last centuries.
So often we judge events based only on their immediate effect upon ourselves. That’s human nature. The Torah, however, takes a longer view and broader view of things.