There is another side to hedonism that is often ignored in the maximization of pleasure, the minimization of pain. Pleasure is frequently transitory, while there are times when one can look back upon a past experience and remember the pleasure and relive it again. Likewise, guilt always will haunt those with a conscious and provide a lifetime of pain.

If our pleasures are purely selfish, the accumulation of guilt can be significant over time. This can certainly dampen daily pleasure.

Desire arising from pleasure is, other things being equal, stronger than desire arising from pain.
--Baruch Spinoza, Ethics, Part 4, Proposition 18

Pleasure is not itself bad, but good. On the other hand, pain is in itself bad.
--Baruch Spinoza, Ethics, Part 4, Proposition 41