I was thinking recently upon the popularity of Nanowrimo, and I asked someone why a month was set aside for writing novels, and not other forms of literature. They replied (aproximately) that the novel was the freest and best way for a writer to express themselves. I responded that that must mean that William Shakespeare and William Blake were never proper writers.
This got me to thinking why The Novel is considered to be the standard form of writing in English speaking cultures today. While poetry and drama (as well as comic books, screenplays, technical writing and erotica) certainly have their fans, it would be hard to make the case that anything but the novel was the Queen of the Literary Arts.
This has not always been true. Literature had been going on for thousands of years before the novel appeared (thus its name). In the West, theology and rhetoric were considered much more literary than novels; and in China, history and poetry had the title.
Why is this? Novels appeared in the West sometime between The Renaissance and The Enlightenment, and appeared in China sometime in what could be described as its modern period. The novel is a reflection of a modern way of thinking. What is a novel, and how does this reflect to "modern thought".
The shortest definition of novel is a long work of prose fiction. On top of this, novels typically have some other characteristics. They generally have episodes that meaningfully build to a conclusion. They usually focus on a single character or a small group of characters, and their interior growth. Their are plenty of novels that are epic in scope, but novels as a form are not about relating fantastic or earth-shaking events, they are about a realistic psychological story.
Let us, then, examine some of these properties. Novels are long because they need to relate a involved yet coherent narrative of a person's life. A short story, while it can carry a very strong image, has a hard time commuicating growth and transformation. Novels are prose because that is the most easy and straightforward way to transmit information. Poetic epics such as the Iliad developed out of the need for a bard to entertain people in a spoken voice. Short, lyrical poetry is good at transmitting sharp images, but not at communicating narrative. Although there are plenty of works in the magical realist tradition, on the whole, prose poetry is also better at communicating single episodes of intense experience, but is perhaps not as well suited at explaining the development of narrative identity. A novel focuses on a single character because the modern, post-modern and contemporary periods all focus on the individual and their psychological development as the basic unit of understanding, instead of the destiny of the tribal group or religious narrative.
The case for novels not being about history is a little shakier. Many novels, such as War and Peace, are about wide, sweeping historical events. However, even when novels involve sweeping events, they are about people, or more accuratly, persons. The nations and trends are not themselves the focus, but how the individual deals with them.
As for why novels are more prestigious than plays, it could do with a nominalist bent in Anglo-American philosophy. A script or play is a causative use of language, in a way, in that language is used to cause other things to happen. Typically, in English and American culture, language is seen as a reflective tool, it is meant to describe things that have already happened, even if they only happened in the imagination.
So all of these are just a brief guess at why you will very rarely see an epic poem or movie script on the New York Times bestseller list.