For some reason, possibly because they think they are obliged to give, many people look away when asked for change. The message this probably sends is 'I couldn't give a fuck about you' rather than 'I feel guilty at my lack of charity'. The best compromise is either:

  1. Look at the person and say "No", but in an apologetic tone of voice.
  2. Give them some change.

Saying 'No, sorry' always suggests (to me at least) 'I have change but I don't want to give it to you' rather than an intended 'I genuinely can't spare my change, but I feel bad about your lifestyle'.