Basically there are two parts involved in breaking the
Vigenère cipher that uses a key that is repeated often while
encrypting. The first is determining the length of the
key. The second is using
frequency analysis on the letters decrypted by the same letters of the key.
To determine the key length first you look for sequences of letters that appear more than once in the ciphertext. The repetition could have been caused by the same letters getting encrypted using the same part of the key or by different letters getting coincidentally encrypted to the same letters. The longer the sequence of repetition the more likely it was caused by the first possibility.
If the repetition we have found is caused by the same parts of the key being used then the length of the key is a factor of the space between the repetitions. The length can be narrowed down by finding additional repititions and finding common factors in the spacing. Doing this we can make a pretty good guess at the possible length of the key.
Now frequency analysis can be used on every Nth letter of the ciphertext where N is the length of the key. So if our guess is that the key has a length of 10 then use the 1st, 11th, 21st... letter and perform frequency analysis on them. Similarly do it also on the 2nd, 12th, 22nd, etc. And so on for each letter of the key.
This technique works better the longer the ciphertext length and the shorter the key length. And once you have identified the key length it still has the same problems that frequency analysis does if the encrypted text doesn't follow normal frequency patterns.
This technique is rendered useless if the key length is the same length as the encrypted message. If the letters of the key are common words the Vigenère square is still breakable. Guessing common words that are probably in the text then trying them at various parts of the ciphertext might reveal possible words in the key. As possible parts of the key are revealed more educated guess can be made at what the keywords are and what the encrypted text is. This technique is more tedious and can be a long process, but it can work.
Finally if the key length is as long as the plaintext message and the letters of the key were selected randomly then we have a One-time pad that is truely unbreakable, but suffers from many problems including key distribution, key generation, and insuring keys are not reused.