There are several different sorts of modal music. In general, the term refers to music that is based, not on a major or minor scale (which is called tonal music), but on one of the other modes, or scales. (technically, "major" and "minor" are modes as well - more on that below)
A few of the main sorts of modal music that commonly get mentioned are:
- Early music, including pretty much everything from Gregorian Chant up to the Baroque era.
- Modal jazz, of the sort pioneered by Miles Davis on Kind of Blue.
- Modern Classical music - I mention this here, though the most distinctive uses of modal music in post-baroque music can be found in the impressionist school, by composers such as Debussy and Ravel. Other more modern composers have also used them, though there's not much that hasn't been at least attempted by modern composers...
- Classical Greek music was also based on these modes, and used pretty much the same names for them, however, the modes themselves are rather different - one shouldn't expect a Renaissance piece in a given mode to have anything to do with the corresponding Greek mode, even though they might have the same name.
- Various other genres of music use their own modes as well - Indian music for example. I won't get into that side of it.
I'll be focusing on the "church modes", as the modes used in Renaissance and medieval music are called. I'll leave their use in jazz and other styles of music to someone more knowledgeable than me.
Before getting into that though, I should mention what I mean by a "mode". Although it's a somewhat loaded term, and carries a lot of extra baggage, for now I'll just say that it's a label given to the collection of pitches used in a piece of music, and also denotes which pitch is the "centre" - typically the one the music starts and stops on, though this varies in different genres. The modes I'll focus on are frequently called "church modes" because of their uses in liturgical music.
What defines which mode you're talking about in a given piece is the order of the intervals in the collection of notes - in other words, if you lay the pitches out in a scale, the position of the semitones and tones determines which mode it is.
As an example, the dorian mode is a series of pitches following the pattern TSTTTST, where T stands for "Tone" and S stands for "Semitone". If you look on a keyboard, and start playing a scale from D, using only the white notes, you get this pattern - i.e. D, E, F, G, A, B, C, D. For each of the 7 note names, there is a mode that can be built on it in this way. Here they are:
In addition to these 7 modes, any of them can be transposed - in other words, retaining the same pattern of tones and semitones, but moving it up by a given interval. Therefore, you can have F Dorian, just like you can have D Lydian or G Locrian.
This system of modes first appeared in Medieval times as a way to classify existing chants, such as the Gregorian ones. They borrowed the names that Greek scholars had used when writing about their music, though the scales themselves are different. Also, the original scheme did not use all the modes listed above. Over the course of the next centuries, the modes became somewhat more standardized into this system, though debates about the proper numbering and ordering of the modes, and their other characteristics raged on until the system was consolidated around 1600 into the system of major and minor scales that we know today. This process was very gradual though - you can see pieces written as early as 1500 that are essentially major or minor, and other pieces (including several by J.S. Bach) well into the 18th Century that retain various modal characteristics.
In addition to the scale used, the mode defined both the range and final note of a given piece. A melody in the dorian mode would therefore typically have a range from D to D, spanning one octave (with maybe an extra note or two at either extreme), and would end on a D. Because of this, D would be called the "final" - this is analogous to the term "tonic" in later music. As well, it would start on either a D, or an A - the A, the 5th scale degree, being the next most important note in the mode. Some melodies had a range going from the 5th to the 5th instead of from the final. These were said to be in a different mode - in this case, a piece in Dorian with a range from A to A would be in the "hypodorian" mode. In polyphonic pieces with several voices, frequently some voices are in the normal version of the mode, while others are in the hypo- version.
There is a huge amount of theoretical debate and disagreement on the subject of modes in Renaissance music, in both contemporary writings (by Zarlino, Morley, and many others) and by modern theorists. To different people in different times, the modes meant many different and subtle things.
In more modern classical music, modes are frequently used to give music a medival sort of sound, or sometimes a folk music sound. Debussy's music frequently has passages in one of the church modes listed above. A very interesting example can be found in Britten's "Ceremony of Carols" in the movement called "In Freezing Winter Night". The movement starts in the Locrian mode, and then shifts mode a few times in the second half.
In general, all this technical stuff aside, modal music is usually used to describe music that is not tonal in the sense that the music from the 18th and 19th centuries is. At the same time, it's certainly not atonal. It just uses a different sort of organization than that used in more mainstream classical music.