We've inherited scores of Midrashim about how the vestments of the high priest can help bring about atonement for various sins. (See the link below for a post containing an excessively long list of them with my commentary)
I say those midrashim are meant literally, and are all examples of correspondence in sympathetic magic, or the belief that one can influence something based on its relationship or resemblance to another thing. Some examples include beliefs that certain herbs with yellow sap can cure jaundice, that walnuts could strengthen the brain because of the nuts' resemblance to brain, that red beet-juice is good for the blood, that phallic-shaped roots will cure male impotence. (Wikipedia)
The midrashim cited below seem right in line with this, and they were written at a time when sympathetic magic was believed to work. So why not?
(Some of you are going to try and read those midrashim allegorically. The Jewish Encyclopedia will explain why:
whenever the literature of a people has become an inseparable part of its intellectual possession, and the ancient and venerated letter of this literature is in the course of time no longer in consonance with more modern views, to enable the people to preserve their allegiance to the tradition it becomes necessary to make that tradition carry and contain the newer thought as well.
Search for more information about ### at4torah.com