The Rambam writes in the last halacha in Hil. Kilayim (10:32) that kohanim may not wear bigdei kehunah (which contained kilayim) when not performing avodah. A kohein who does so is liable for malkos. Ra’avad disagrees and writes that kohanim may wear bigdei kehunah even if they are not engaged in avodah so long as they remain in the mikdash. I don’t have time for a full analysis of the machlokes, but do want to point out a Midrash Rabbah on Parshas Chukas which comments on Aharon’s death:
והלא אם יצא כהן גדול בבגדי כהונה חוץ מהר הבית, סופג את הארבעים, שהם צמר ופשתים, אלא להודיעך, שבלשון שקרבו לכהונה ואמר לו: קח את אהרן, בו בלשון אמר לו לעלות ההר
In finding fault with Aharon wearing the bigdei kehunah outside the mikdash, the Midrash seems to clearly side with the Ra’avad. What would the Rambam make of this? Without getting into the details, one method to defend the Rambam (see Ha’amek She’eilah to She’ilta 126 - p. 13 in the standard edition) would be to prove that the Midrash is an isolated view which the gemara disagrees with. I'm putting this one on the back burner for another time.