A repertory theatre troupe, often shortened to simply 'repertory', or 'rep', was historically a theater troupe that put on classic plays (i.e., ones that did not require them to pay royalties and that were familiar to most people) in short rotation, generally putting on a new play every week or two. During the height of their popularity most towns could support a repertory theater, although later on it become common for a single repertory troupe to travel a set circuit through multiple local towns.
Repertory theater was quite popular in the UK from the 1910s until the 1950s, and was not uncommon in other countries. There are a number of modern repertory companies and theaters in the UK, America, and Canada, among other places. The Birmingham Repertory Theatre (AKA 'The Rep') is perhaps the most recognizable of these, although there are many others that are still well-known and going strong. However, these days repertory companies are much less likely to be a fixture of the average person's entertainment.
Historically, most troupes would put on a new play each week, including nightly performances and matinees, while at the same time rehearsing the next week's play. The quality of the performances was generally not what one would expect of a theater group that stuck with a performance for an entire season or longer. Because of this repertory theater was not generally seen as particularly respectable or high-brow, but it was good fun for the audience and provided a paying gig for the actors, and acted as a springboard to start many famous actors' careers.
Because the repertories were a reliable source of entertainment, always coming to a given theater at the same times each week, they could expect to pull in regular crowds and provide a steady income to their actors and crew. They were generally for-profit organizations, and this encouraged them to put on plays that would draw a crowd -- which meant familiar, popular plays.
Modern reps -- which are more and more frequently called 'reps' rather than 'repertory theatre' -- are very often non-profit organizations. For example, the well-known Stratford Shakespeare Festival, the Barter Theater, the Oregon Shakespeare Festival, and the Milwaukee Repertory Theater are all non-profit organizations that are generally considered to be repertory theaters. This, in part, has been responsible for many troupes becoming more open to producing new works than in the past, although obviously Shakespeare and other classics are still quite popular. Modern troupes generally have a longer 'season' of three to six weeks, which results in more rehearsal time and, hopefully, higher quality shows.
While these are not formal terms, a troupe that travels is more likely to be called repertory, while one that stays in one theater is often called a stock company. Many repertory theaters have both stock troupes and allow travelling troupes to perform, resulting in a bit of confusion about the terms. A repertory may be contrasted to a combination company, a travelling company that puts on only one play in a season.