acharayus for bikurim - until what point?

Rambam (2:20) holds that someone who brings bikurim which then are lost/destroyed/became tamei is chayav in achrayus unless the bikurim arrive intact to har habayis. Why har habayis – the mitzvah of bikurim is to bring them into the azarah (see Radba”z)?! R’ Chaim explained (as quoted in R’ Reichman’s “Reshimos Shiurim” on Sukkah) that the din of achrayos is not a function of insuring the kiyum mitzvah of bikurim, but is a separate din that one must attend to the bikurim, which has an independent shiur until one reaches har habayis (he brings a makor in Toras Kohanim for such an idea).