Short answer: If HE has custody of it, it is NOT yours. You have transferred it to him. If you want him to ship HIS firearm to you, it needs to be done thru a FFL, and you will be required to do a transfer at that FFL just like any other buyer.
There is no such thing as "loaning" or "storing with a friend" in the firearms laws. When you left it in the custody of another person it was legally the same as selling it to him (from a firearms law standpoint).