An ugly person's physical appearance matters less the more one interacts with that person. This holds provided the person is not annoying, wicked, unpleasant or any of the other negative human behaviors that make people disliked.
If a beautiful person is an arsehole, their appearance becomes less attractive as one interacts with them.
Thus, it seems both beauty and ugliness diminish with time and bad behavior.
On an unrelated note, Urban Dictionary defined inner beauty as "something invented by ugly girls to make themselves feel better about themselves". While I do not agree with the statement, I have always found it funny. It is also the basis for a sexist Nigerian joke (which I also find very funny). The joke is that "some men have inner money the same way some women have inner beauty".