The X server is a key component of the X Window System which hosts displays. X clients connect to the X server and send it data, and the X server displays the data on the appropriate monitor. The X server also gives X clients things such as keyboard and mouse input, or any other information the client wants. The communication between the two is done through The X Protocol, over TCP or UNIX domain sockets.
Some stupid people argue that an X server should be called an X client. These people are not so bright: it's obviously a server because it has the listening socket. The clients, or programs, connect to the X server, thus are clients.