"Only fond of the leftover peach" is a famous Chinese four-character idiom, or chengyu (成語), which serves as a euphemism for homosexuality.
In Ancient China, homosexuality was not particularly frowned upon, and it was quite common for rulers to count men as well as women among their sexual favorites. One of the most famous of these male concubines was Mizi Xia, the sexual favorite of a certain Duke Ling of Wei during China's Warring States Period (ca. 400 BC). According to a famous collection of anecdotes known as The Garden of Stories...
Mizi Xia became the favorite the Lord of Wei. According to the laws of Wei, driving the Lord's carriage without authorization was punished by amputation of the feet. When Mizi Xia's mother became ill, someone heard about it, and slipped into the palace at night to inform him. Mizi Xia forged an order from the Lord, took the Lord's carriage, and went to see her. The Lord heard about it, but only thought him more worthy, saying, "How filial! For the sake of his mother, he committed an act which could have led to the loss of his feet!"
Another time the lord was relaxing in his orchard. Mizi Xia ate a peach and found it sweet, so he did not finish it, but instead offered it to the Lord. The Lord exclaimed, "He loves me so much that he forgets how good it tastes!"
But when the time came that Mizi Xia's looks faded and the Lord's love slackened, he became offensive to the Lord. The Lord said, "This man once drove my carriage under false pretenses, and what is more once fed me a leftover peach!" Thus, although Mizi Xia's behavior had not changed from before, the reason his actions were first considered worthy and later gave offense was because the Lord's love had changed to hate."
From this famous story arose the phrase "fond of the leftover peach" as a euphemism for a predilection for male-male love. But this phrase was only three characters long, and the Chinese especially love four-character phrases, so an "only" was added in reference to exclusively homosexual preferences (as opposed to the bisexuality practiced by the lords of Mizi Xia's day), and now that it had four characters, the phrase really caught on.
Translation from The Garden of Stories by yours truly.