The mutual authentication problem is for two parties who share a secret to each prove their knowledge of this to each other. For instance, with fighter jet friend or foe systems it's vital that neither of two friendly craft attack the other; but to err on the side of caution would be devastating if a foe were encountered.
In authentication I set out a protocol, the challenge-response system, whereby one secret holder may prove to the other that they hold the secret, but in closing that writeup I claimed that running challenge-response in both directions won't suffice for mutual authentication. The reason for this is that it gives rise to a reflection attack as follows.
Suppose our fighter jet F1 is running the protocol in both roles, that is, issuing challenges and responding to challenges issued. It does this on the basis of a secret key K which is shared between all other friendly jets Fi. Certainly then if a friendly jet F2 is challenged with a nonce N it can respond with {N}K and F1 knows to trust F2; meanwhile, F1 will respond to F2's challenge of M with {M}K and thus ensure that F2 trusts it.
So multiple use will identify friendly craft as such. The problem is that a hostile intruder I (probably very hostile, given that it's an unfriendly fighter jet) can also pass itself off as a friendly by careful choice of its challenge- specifically, it reflects F1's challenge N back to F1, who will respond with the correct encryption {N}K. The hostile can then use that as its response to F1's challenge, thus convincing F1 that it knows the secret key K.
Or, in more abstract form:
Challenge-response reflection attack
F1 → I(F2): N
I(F2) → F1: N
F1 → I(F2): {N}K
I(F2) → F1: {N}K
This problem can be solved by extending the 'conversation' to accomodate challenge and response from both parties. In fact, this is possible with only three transmissions, less than the four needed for two runs of challenge-response, since the second challenge can be bundled into the first response:
ISO 9798-2 three-pass mutual authentication
1) F1 → F2: N1
2) F2 → F1: {N2,N1,F1}K
3) F1 → F2: {N1,N2}K
Here we have multiple messages bundled together then encrypted into a single long ciphertext; {N2,N1,F1}K is different from {N2}K,{N1}K,{F1}K the bundling together of three shorter ciphertexts. An eavesdropper would still be unable to extract N2 or N1 from the latter without the key K, but it would be no trouble at all to re-order the observed messages and send back {N1}K,{N2}K. Thus the protocol cannot be based on such exchanges and this does increase the cryptographic (and hence computational) complexity somewhat. By the end of step 2, F1 is certain that F2 is to be trusted, as only a friend could have encrypted together F1's challenge and name. F2 also includes a challenge in this bundle which F1 responds to, including his own challenge to confirm the session and encrypting to confirm identity as a friend of F2.
F1 will still have to be running the protocol in both roles since there is no way to establish who acts as the initial challenger; F1 could receive a challenge nonce from F2 rather than being the initiator, and thus must be able to perform step 2 in response. However, the inclusion of target names prevents the reflection attack: suppose our intruder I is attempting to impersonate an F2 and issues his own challenge after being challenged by F1:
1a) F1 → I(F2): N1
1b) I(F2) → F1: N1
2b) F1 → I(F2): {N2,N1,I(F2)}K
2a) I is now stuck, as he needs to respond with {N2,N1,F1}K but does not have this message.
Had the protocol only specified a response of {N2,N1}K at step 2, then the intruder would have received this in step 2b and could have reflected it for step 2a; thus this inclusion is essential to the protocol. Under the assumptions of a shared secret, secure cryptography and the ability to encrypt messages together, the three-pass protocol is believed to be secure; that is, no attacks have been found! Of course, the base assumptions may themselves be impractical- as usual, the shared secret is the most likely sticking point, as it requires prior arrangement or, for previously unknown participants, some form of key exchange.
Reference: Formal analysis of Cryptographic Protocols Taught Postgraduate course, Laboratory for Foundations of Computer Science,
University of Edinburgh.