A gentlemen's agreement is an agreement, usually unwritten, guaranteed only by the honor/pledge/solemn word/secret understanding of those involved. It depends on the participants being men of honor, or nowadays, people with honor.
In my experience, such an agreement between "gentlemen" is often an agreement to tolerate ungentlemanly behavior, whether explicitly ("We won't tell on anyone") or implicitly through the threat of force should someone point out the agreement has been broached. I suppose in the past it might have been a different affair, when honor was more of a concern.