The following
dilemma may be posed to you at the beginning of
Baldur's Gate II:
Suppose that an evil wizard has captured you and someone else and
locked you in separate rooms, each with a button. You can either
press the button within the next hour or not. If you press the button
and the other person doesn't, they will live and you will die. If
they press the button and you don't, you will live and they will die.
If you both press the button, or if neither of you presses it, you
will both die. You do not have a chance to discuss your choice with
each other beforehand. What do you do?
In Baldur's Gate II, the other person was your sibling, but it's
slightly easier to analyse if you assume that they are a random
stranger and you are indifferent to whether they live or die. Then
you might think that you should not press the button, because if you
do, then you will die no matter what the other person does. But if
they think that way as well, you will both refrain from pressing your
buttons and you will both be killed. On the other hand, if you both
decide to be heroic and lay down your life for the other one, you
will still both be killed.
We can look at this using the concept of super-rational thinking
described in Metamagical Themas. Super-rational thinking means that
you reason:
"There is a right thing to do here (as in maximising the expected
value of the result, not as in ethical). I don't know what it is
yet, but because the other person and I are both reasoning beings,
and the situation is symmetrical (nothing changes if you swap our
roles), I know that we'll come to the same conclusion about what it
is."
That doesn't seem to work here, because if you both do the same
thing you get the worst possible result. That's because we've ignored
the possibility that rather than choosing to press the button or not,
you might use a randomising device and elect to press it with
probability p. Then, given that you both use this strategy, your
chance of survival is p(1-p), which is maximised when p=0.5.
Therefore if you are both focused on your own survival, you would do
best to flip a coin. This is also the best strategy if you each believe
the other person's survival to be infinitely more important than
your own. It even works if you both think something like "I value her
life more than mine, but I am indifferent between a 90% chance of
survival for her and a 100% chance of survival for me", so long as
it's the same something and you both know it's the same
something. (If the other person is your enemy, you want her to
die and you know she feels the same way about you, you can still apply
super-rational thinking but the results will be different.)
Unfortunately, this strategy breaks down if you value her life more
than she values yours or vice versa, because then the situation is no
longer symmetrical. You might also wonder whether she knows about
super-rational thinking - it doesn't work unless you both do it.