A much better phrase than 'one-horse town', but meaning the same thing - a one-eyed town is a very, very small town.

Basically, it goes like this... people knew that inbreeding caused mutations. A popular belief was that the inbred often had their eyes closer together (facial features being a popular tool for determining intelligence back then...) so if the residents of your town only had one eye, well, they must have really been forced to inbreed.