It's just the result of a
natural progression of improvements made to a basic primitive
Turing State Machine. First a standard set of
instructions was laid down, then
expanded as more people started programming microprocessors. This pretty much culminated in the
x86 chips, which had a whole host of random
machine code commands that you could give a microprocessor. When
compilers started to become an
industry standard, the microprocessor actually took a step backwards in the form of the
RISC chip, which pared down the instruction set to about 10 commands that could be executed at
blazing speeds (most compilers couldn't take advantage of the
CISC architecture that was in the x86).
To take advantage of RISC speed but retain backwards compatibility with the x86, the Pentium chip acts as a black box with respect to processing x86 commands, but actually interperets these commands in hardware and translates them into instructions which it feeds into an on-board RISC processor. Transmeta's Crusoe chip takes this concept a logical step farther, and uses software and/or EEPROM to allow the developers (currently, only Transmeta developers) to define any sort of instruction set they want, x86, Java byte-code, etc, which can then be translated into the native instruction set of the chip during execution itself.
I think the Crusoe also uses a VLIW architecture, which means that some commands can be executed concurrently on the chip, as long as they are determined to be independent processes.
During the development I kind of whipped through above, manufacturing processes have allowed for construction of smaller components and leads, which has meant that microprocessers have also gained the ability to support longer instructions. Before the Pentium, most computers either used an 8-bit or a 16-bit instruction width, which sometimes required several cycles to completely load a single instruction. The Pentium introduced 32-bit instruction width to the mass market. Today, VLIW processors can load 64-bit or even 128-bit 'instructions' at a time, which are actually two, four, or eight instructions in parallel.
All of the above is from memory. Please message me if i have a factual error somewhere.