A program is closed source if the source code the program was built from is not made available. This is not necessarily the opposite of 'open source', as the term 'open source' is nowadays synonymous with 'Free Software', and usually implies the additional right to freely redistribute the program. A program can be 'closed source', while being freely redistributable - such a program is often called Freeware, or public domain. A program can also provide access to the source without necessarily being freely redistributable, usable, or modifiable (the original meaning of the term 'open source').
It is often argued that closed source software is somehow 'more correct', as the user is prevented from modifying it. This argument is bunk, however, as releasing software as closed source prevents the user from performing their own validation, but does not prevent the user from modifying the program.
A program can prevent redistribution or modification without hiding the source code, however. POVRay and Pine are both distributed under licenses that allow the program to be either distributed or modified, but not both. In effect, they are freeware programs, with the source available for inspection.
The most famous 'open source, but not Free' program is PGP1. PGP's source code was made available to the general public, under a license that allowed it to be compiled, but not used. To use PGP, one had to purchase a commercial version of it, or download a Freeware version. This allowed the users access to the source, without giving the the right to use the program, or make their own programs based on it.
Access to the source means that anyone, not just the (relatively) small group of developers, can perform whatever testing is required to prove to their satisfaction that the application is correct. If necessary, they can spend more time and money testing it than the developers have. And when it's all done, they can compile the application the same way the developers did, and confirm that the program they've been shipped is the same as the one they validated.
The reason why PGP was so successful was that it could be proved to the satisfaction of notoriously paranoid cryptography experts that the program was correct and that there was no malicious code hiding in it. At the time, no other cryptographic program offered this2, and thus for many, no other program could be trusted with their secrets.
Not providing source code does not prevent the user from modifying a program, it merely makes it harder. Even encryption and obfuscation of the object code will not stop a determined programmer. Executable wrappers like safedisk, which employ state-of-the-art anti reverse-engineering methods are cracked within days of each new release. They are thwarted by the simple fact that if you want your program to run, it must at some point unencrypt itself. The emulator Callus was maintained for over a year without access to the source code - the developers dissapeared off the face of the earth, but this did not stop third parties adding new features to the raw object code. If you're really bothered by people modifying your program, license it under terms that forbid modification, then sue anybody who does.
As well as the problem of having to take the word of external developers that the code you are using is correct, closed source software has additional complications - there are very few owners of the original source, making it not impossible that all copies of the original source may be destroyed or lost. Even if the source is still available, the owners may refuse to perform maintainance of the software, particularly if it is reaching the end of its lifecycle. To avoid these problems, many closed source solutions are now provided with the source code held in escrow by a trusted third party, to be delivered to the client should the developers go bust, and to help ensure that the developers do not lose all copies of their own source code.
1 - PGP is no longer released under such a license, after being bought by Network Associates.
2 - Now there are lots of Free crypto programs - GPG, a Free reimplimentation of PGP, is the most well known.
'Open source' does not mean 'freely modifiable', and 'closed source' does not mean 'not modifiable'. To assume anything else is to risk humiliation when your 'unmodifiable' copy protection is cracked, or your trade secrets are reverse-engineered. Relying on closed-source alone to protect against liability should your customers modify the software is a grave mistake, as closing the source is not actually sufficient to prevent them doing so. It makes no difference to the legal position of either the vendor or the customer whether the customer modified the software by altering the source and recompiling, or by disassembling the closed-source binary. Modification and reverse-engineering are legal matters, and should be prevented by the license agreement, rather than being hindered by obfuscation. Intellectual property should be protected by copyright and patent law, rather than the hope that no-one will bother to reverse-engineer your object code.
If closed source has any role in enterprise software, it may be in addition to an NDA in keeping trade secrets secret, or as an additional safeguard preventing patented or copyrighted algorithms less likely to be copied, but it certainly cannot do these things in and of itself. Closed-source may remain a useful tool in forcing consumers to stick to license agreements, but even this is becoming a lost cause. Research into copy protection is becoming more and more costly, and copy protection is becoming less and less effective.