Nessus is a security scanner, written primarily by Renaud Deraison with assistance from Michel Arboi and the usual cast of thousands. It is available under the GNU GPL at http://www.nessus.org. Presumably Nessus is named after the paranoid alien from Larry Niven's Ringworld novels, but I didn't remember to ask the one time I met Renaud in person, and it seems like a funny thing to email him about. The first version was released in 1998, and currently Nessus is considered in the top tier of security scanners, and certainly the best open source one. Commercial tools like ISS or CyberCop might offer a few more features, but most people decide that getting those extras isn't worth the thousands of dollars they would cost.
The basic function of Nessus is to scan a machine, determine what services are running on said machine, and then try to figure out if any of those services are vulnerable to known attacks. Nessus can carry out the scanning itself, using nmap, or you can choose the list of target IPs and ports yourself, which is useful if you want to use a more advanced scanner like sscan, unicornscan, or scanrand. Like it's namesake, Nessus is fairly paranoid. In particular, you can't fool Nessus by running a service on a different port, or by wrapping it in SSL; this is because for each open port, Nessus examines what responses it gets back to various inputs, and attempts to determine what kind of service is running, regardless of the port. Nessus will then attempt to fingerprint the service down to a specific product and version, if possible. This is a distinct contrast to many other scanners, which make assumptions about port numbers. For example, running a telnet server on port 80 will fool some scanners into thinking you are running a web server, and then it will waste time trying to execute Apache exploits against your telnetd.
Once Nessus has figured out what is running where, it will start testing each service for problems. It does this through a (large) set of scripts written in Network Attack Scripting Language (NASL), a Nessus-specific language which is easy to write network attack code in. The set of NASL scripts available for Nessus is large and growing. Often, shortly after a new vulnerability in a product is announced, a NASL script to test for that vulnerability will be released. Nessus includes both safe and unsafe checks. Safe checks will just try to figure out the service's version number and check that against known vulnerable versions. The unsafe checks are much more reliable (as they actually attempt to exploit the service), but in most places dropping a production web server is considered a big no-no, so usually people scan with safe mode on, and deal with the false positives later.
In addition to testing using NASL scripts, Nessus has the ability to run external testing and attack programs like Nikto and Hydra. There is also some amount of framework for writing plugins in C, but it's generally only used when NASL can't handle the situation for whatever reason.
Once Nessus has completed it's testing, it produces a report which can be saved out in various formats, such as HTML, LaTeX, XML, or a native Nessus format.
Renaud is commercializing Nessus through a company called Tenable Security, based in Columbia, Maryland. Among their products is a Windows port of Nessus and various "Nessus appliances", which you basically plug into a network to do scanning for you.