I deliberately don't give a lot of physical description of my characters. I try to give enough to help get readers started on creating their own mental image of my "people". I've always believed the main trait (honesty, courage, stubbornness or whatever) is far more important in creating a character that works.
However, that doesn't mean your characters shouldn't judge other characters by their appearance. After all, human frailities are reflected in fiction so why shouldn't this one be? I could see comic, as well as tragic, stories coming out of characters getting it wrong when judging those in the story with them. And even when a character judges another correctly, what are the consequences of that?