There are many good reasons to believe that the Gospel of Mark was the first gospel to be written. (In this case, "priority" refers to being chronologically first, as opposed to being more important or somesuch.) Markan priority is one key to understanding the Synoptic Problem, and, by extension, the history of the development of the New Testament.

The arguments go something like this:

1. The argument from omission: Mark is significantly shorter at 661 verses than both Matthew (1068) and Luke (1149). This could mean one of two things: either Matthew and Luke expanded on Mark, adding things like a birth narrative and various speeches from Jesus, or, Mark compressed Matthew and Luke, removing all that other material. However, the former option seems much more likely. This is because...
2. The argument from length: Individual stories in Mark tend to be longer than their parallels in Matthew and Luke. To choose just one example of many, Mark takes twenty-one verses to tell the story of the Gerasene demoniac (5:1-21). By contrast, Matthew takes seven (8:28-34) and Luke takes fourteen (8:26-39). A brief skimming of Mark's gospel, even in translation, is enough to demonstrate that his style tends to be pretty rambly, to the point of being repetitive and redundant in places. If Mark truly had the desire to cut a longer text down, then why would he make his individual stories so much longer than the parallel versions?
3. The argument from diction: Mark uses street-level Greek, containing a lot of colloquialisms, tense changes, and careless use of grammar. For instance, he uses the word krabattos in 2:4, which is a slang word for "mattress" or "cot." Matthew and Luke quietly change the word to the more proper kline, "bed," in 9:2 and 5:18 respectively. It's hard to imagine Mark taking Matthew and Luke's perfectly good Greek and deliberately making it worse (though given the way that my students occasionally mangle their assigned readings, perhaps it's more likely than we think!).
4. The argument from Aramaic: Mark uses seven Aramaic expressions, which he translates into Greek for the benefit of his reader. Matthew uses only two, and Luke uses none at all. It is easier to imagine Matthew and Luke removing the foreign language passages than it is to imagine Mark adding them in.
5. The argument from difficulty: Mark depicts Jesus in ways that are often unpalatable to a "nice" Christian view. For example, Jesus' family members bring up the possibility that Jesus is crazy in 3:20-21, a passage that appears neither in Matthew nor in Luke. In Mark, Jesus is frequently portrayed as losing his temper, for example in 3:5, where we read "He looked around at them with anger." In Luke 6:10, the anger disappears from the "look." In Matthew 12, the verse is omitted entirely -- though the evangelist does preserve the verses preceding and following it. Needless to say, it is much easier to imagine Matthew and Luke changing Mark's difficult readings than it is to imagine Mark putting them in where there were none before.

All of this is, perhaps, circumstantial evidence, but nevertheless it builds a very strong case. This is somewhat distressing to certain literalist Christians for whom it is important to believe that the gospels were written by eyewitnesses. According to tradition, Mark was not a disciple of Jesus (according to Papias, a second-century Christian writer, Mark was Peter's "interpreter"), while Matthew was a disciple. It seems odd for a disciple to copy his stories about Jesus from someone who did not know Jesus directly, does it not?

There is a small minority of scholars who continue to defend Matthean priority even in the face of the arguments laid out above. Not all of this work is bad by any means, but the majority of scholars find it suspicious due to its apologetic nature. Besides, it is unlikely that any of the gospels were written by the people after whom they were named anyway. Therefore, even on the off-chance that the theory of Matthean priority develops a strong defense, that doesn't get us any closer to the Historical Jesus.

Further reading:
There is a good summary of the arguments for Markan Priority on Stephen Carlson's excellent Synoptic Problem web site:
http://www.mindspring.com/~scarlson/synopt/2sh/#Mark
For more on the synoptic problem see:
http://www.hypotyposeis.org/synoptic-problem/