X is a memory hog

Fact: X servers use memory mapping. ps and top are accurately reporting how much memory X is addressing, but most of that memory exists on your video card. If you have a 16MB video card and top reports X is using 19MB, the X server itself is actually using 3MB of your system's main memory, not 19MB.

The other issue in play is that X servers are storing data from X clients, or programs that use X. That means that some of the memory X is allegedly "hogging" actually originate from other programs you're running. Naturally, as the number of programs running goes up, so does memory usage. I don't think anyone can debate that, let alone blame it on the windowing system.