The "tritone" (interval spanning three whole steps that characterizes the sound of what we call the "locrian mode") was called "diabolus in musica" ("The Devil in Music") in the middle ages. I like to think it's because it has a sort of spooky sound to it, but those people lived a really long time ago and had never heard music played in harmony as we know it. Certainly they hadn't seen the Exorcist, and though their daily life could be more horrible at times, surely it was not accompanied by a spooky sounding musical soundtrack everywhere they went. So it seems reasonable to me to assume that they had not developed a cultural response to that sound as being "demonic" or "satanic" or whatever. Probably the "devil" in "diabolus in musica" is more like the idea of "defect" than "evil." This is because it does not function melodically or harmonically the way a "perfect fifth" does, which is the interval which occupies that position in all the other modes. As a result the mode we call locrian was not part of the system of musical modes as "officially" formulated in the middle ages, but because of the layering of two separate ideas, actually came out in practice under the guise of the "hypophrygian mode."

They had two basically unrelated ways of thinking about the organization of musical notes into what we would call "scales." The first was similar to our idea of modes, overlapping sequences of notes, each with a particular central note (called the "final" in their system). The final is the note on which the melody ends, and it forms a central sort of gravitational hub around which the melody revolves. To say a melody is "in the Dorian mode" is like saying a concerto is written in "B major" in modern terms -- just as B is the main note in B major, D is the main note in the Dorian mode. Each mode also has a designated "dominant" which is supposed to be of secondary importance in organizing the melody.

It went like this:

  MODE              THEORETICAL SPAN ("AMBITUS") 

dorian                 (D) e  f  g (a) b  c  D
hypodorian     a  b  c (D) e (f) g  a
phrygian                  (E) f  g  a  b (c) d  E
hypophrygian      b  c  d (E) f  g (a) b
lydian                       (F) g  a  b (c) d  e  F
hypolydian           c  d  e (F) g (a) b  c
mixolydian                      (G) a  b  c (d) e  f  G
hypomixolydian          d  e  f (G) a  b (c) d

(FINAL) of each mode
(dominant) of each mode

They distingished modes by "authentic" (dorian, etc.) and "plagal" (hypodorian, etc.), depending on the ambitus.

This system of modes was adopted very loosely from ancient Greek music theory. By the time they created this in Europe the middle ages they no longer had any real cultural connection to the Greeks and based it simply on the writings of Greek theorests, and of course the result though superficially resembling the modes of Greek music, represented a very different musical tradition in practice.

The second idea that got overlaid on the very abstract idea of the modes was the concept of "hexachords." Hexachords are groups of six notes that are similar to the first six notes of our major scale. There were three kinds of hexachords, "natural," "hard" and "soft." Within each hexachord, the notes were named "ut, re mi, fa, sol and la, respectively. (In later usage "ut" was changed to "do" in Italy so all the notes would start with a consonant, and "ti" was added to accomodate the seventh note of the scale which did not exist in the hexachord system. "Do" is still called "ut" in France.)

NATURAL   c   d   e   f   g   a
          ut  re  mi  fa  sol la   

SOFT                  f   g   a  (b)  c   d
                      ut  re  mi  fa  sol la

HARD                      g   a  (B)  c   d   e
                          ut  re  mi  fa  sol la


Depending on which hexachord it belonged to at the moment, most notes could represent various steps (a could be la, mi or re, c could be sol, fa or ut, etc.) Hence they named the notes "a la-mi-re" "c sol-fa-ut" etc.

B is a special case. In order to avoid the defective tritone (F-B), it appeared as what we call b-flat (b) when it was fa in the soft hexachord, and as b-natural (B) when it was mi in the hard hexachord.

So you might have a melody that starts out in the natural hexachord and then moves into one of the others and back, such as this melody in the dorian mode:

d   c   d-d   e   f   d   e   f   g   a   f 
re  ut  re    mi  fa  re  mi  fa  sol la  fa

g   a-a-a   a   g   a   c   (B)   a-a-a-a
sol la      re  ut  re  fa   mi   re

(single letter = one beat, double = 2, triple = 3, etc.)

The point where "a" changes from "la" to "re" is where you move out of the soft hexachord into the hard.

Sometimes you could scoot up to a b-flat for one note, leaving from "a" and returning to "a", and this was not considered a true change between hexachords. They had a special name for this procedure, which was "fa supra la." For example:

f   c   d   d   a   (b)  a-a-a  a   g   f   
fa  ut  re  re  la  fa   la     la  sol fa  

g   a   a-a-a
sol la  la
So in the same snippet of melody both b-flat and f appear as "fa," even though technically the use of b-flat would not have been considered a change from one hexachord to the other.

Now this is where the whole "locrian" thing comes in and why I posted this in conjunction with the locrian mode node. Even though what we term the locrian mode in modern usage did not exist in medieval theory, because of the overlapping of the modes and hexachords and the use of fa supra la, there are dozens of hypophrygian melodies from the middle ages that actually use a tonality we could only understand as locrian, as interpreted purely from the point of view of modern modal practice.

Even though the theoretical ambitus of the hypophrygian mode is b c d (E) f g (a) b, there is no way to get down to the low b without changing to the hard hexachord, which you can't do without using the g and a below it, which are out of range. Likewise, the high note b usually appears in the hypophrygian as fa supra la (and sometimes though less often as part of a complete migration into the soft hexachord) because to change to the hard hexachord and therefore get to b natural, composers would typically want to use the note c and come down on top of the b. Thus the true range of hypophrygian is

c d (E) f g (a) b-flat,

which is unmistakably what we would call the e locrian mode. Here is a sample of such a Gregorian Chant melody:

f d e f g a a (b) a g-g f-f, a (b) a g-g e-e, g a g f e f-f g f-f e-e


One more footnote -- in the actual hexachord system there were lots of overlapping hard, soft and natural hexachords spanning a wide range, like this:

G a b c d e                      (Hard)
      c d e f g a                (Natural)
            f g a b c d          (Soft)
              g a b c d e        (Hard)
                    c d e f g a  (Natural)

etc.

The lowest a and b are called "a re" and "b mi" respectively, since there are no other overlapping hexachords and they can therefore only function as one note each. The c near the bottom could only be fa or ut, hence "c fa-ut," whereas the next c is "c sol-fa-ut." The lowest G, since it is the first note of this big complex system, had a special name -- instead of just calling it "G ut," it was written with the Greek letter Gamma and known as "Gamma Ut." From this, the whole system of overlapping hexachords came to be known as the "gamut," which we still use in expressions like "running the whole gamut."