At the threshold of falling prey to temptation at the hands of Eishes Potifar, Yosef sees the image of his father (Rashi based on Sotah 36) and it is this image which causes him to desist from sin. The Yerushalmi (Horiyos 2:5, 10a in Vilna ed) adds according to anothe view that he saw the image of his mother.
It's a simple point, but one I think that is worth spelling out: It's not a halacha in the Rambam or a law of Egypt or even a sense of morality that causes Yosef to pause. All these are but minor obstacles that the yetzer hara can easily surmount. What grabs Yosef's attention is the image of Ya'akov the tzadik, Ya'akov his father. We ship our kids off to school and shul to learn and memorize rules and regulations, do's and don't's, but when push comes to shove, it's not a memorized rule that will guide their behavior -- it's dmus d'yukno of Aviv v'Imo. It's the image we present as parents that will stick in our children's minds.
What is the significance of the Yerushalmi's insight that Yosef saw his mother's image? My wife suggested a pshat that beat any idea I had. Midrashim indicate that Eishes Potifar saw in the stars that she was fated for Yosef. There was truth to this vision, as it was her daughter who became Yosef's wife. Seforim also compare Eishes Potifar's desire for Yosef with that of Tamar for Yehudah -- both had the purest motives in wanting to be with a tzadik and bring his children into the world. Yosef could easily have justified succumbing to her advances as a surrender to the inevitable dictates of fate. Yet, he remembered the image of his mother Rachel, who was fated and destined and desired becoming Ya'akov's wife, yet gave all of that up for the sake of Leah. Rachel's heroic action for the sake of heaven against what fate decreed was the impetus for Yosef's own heroic resistance.