The Sifsei Chachamim quotes an explanation from the Maharashal: you can’t wear gold and silver jewelry with the beggar’s clothes of a slave. When Bnei Yisrael were told to take gold and silver, even though they had not been told to do so, they asked for clothes to go with those accessories.
When Rashi says the garments were the most significant item taken, he is speaking from the perspective of the Egyptians. The Egyptians were able to delude themselves into thinking they were giving out their gold and silver jewels as a loan, but would get them back. However, the Egyptians knew that the Jewish people wore distinctive dress that separated them from Egyptian society. They knew that a Jew would only wear Egyptian clothes after doing extensive alterations [try going to a department store and finding something your wife or daughter can wear off the rack]. They could not even pretend that they would ever see the same garments again, and therefore, giving away their clothes was their most difficult sacrifice.
The Ksav Sofer says a chiddush nifla. The reason Bnei Yisrael asked the Egyptians for new clothes was l’shem mitzvah, because they wanted to have special clothes for Shabbos and Yom Tov. It was their appreciation for the mitzvah of kavod Shabbos which make those garments even more valuable than gold and silver.
The Torah tells us that the mitzvah of Shabbos is “zecher l’yetzi’as Mitzrayim,” a way of remembering leaving Mitzrayim. Ksav Sofer writes that on each and every Shabbos in the desert the Jewish people dressed in these same clothes that they took from Mitzrayim l’kavod Shabbos and they would think back and remember these miraculous moments of leaving Egypt.
(Maybe this is why the Torah commands that the korban Pesach had to be eaten with “na’aleichem b’ragleichem,” your shoes on your feet. I don’t know about you, but I wear casual walking sneakers during the week, but on Shabbos put on less comfortable but more dressy wingtips. Bnei Yisrael had taken a full wardrobe, shoes included, from the Egyptians. They must have been tempted to dress up for the seder, including putting on their new dress shoes l’kavod Yom Tov. The Torah advises not to do that. Wear “na’aleichem,” your shoes, the old pair of sneakers that you broke in already, not the stiff wingtips that still hurt your feet that you took from your Egyptian neighbor. The korban Pesach had to be eaten ready to run out the door, and you can’t do that in leather-soled dress shoes.)