Can a mortgage company make you pay for an appraisal even if you do not close on the house?

I was in a contract for a brand new home when it was discovered that the exterior brick was sub-standard by a licensed real estate inspector hired by me. I was informed by the builder that the brick was made that way and was warrantied by the brick company. The builder neither agreed or disagreed that the brick was sub-standard but stated that he was not replacing all the brick. I refused to close and he agreed to return the earnest money. (I haven't seen it yet) I was then informed by the mortgage company that I am still responsible for the appraisal even though I'm not accepting the house. My question is shouldn't the builder have to pay for the appraisal since he was the one that defaulted on the contract?