Plus fours are a type of short trousers that extend four inches below the knee (hence the name). While short pants have been worn in Europe for centuries, plus fours came into fashion amongst the athletes of England in the early 1920s, who needed leggings that wouldn't get caught on protrusions or catch in bicycle gears, but that would still look spiffy. The plus fours were most likely a compromise between long shorts, which were typically worn by children, and full-length trousers, worn by adults.
Plus fours were the most popular of the fashionably short trousers, but plus twos, plus sixes, and plus eights were also common. Today any of these will likely be called plus fours, because, quite frankly, nobody cares. Plus fours are often made of tweed or tartan, and while they are not formal wear they are by no means shabby dress, and were often worn as casual wear by the upper class, including the royal family.
Over time plus fours became associated with golf more than any other sport, largely because pro golfer Payne Stewart favored them during the time he was dominating the American golf scene.