Fire ships were used, from the time of
the Byzantine Empire up to the 19th century, to disrupt, scatter, and sometimes
destroy enemy fleets. They were usually employed by badly outnumbered and besieged
port towns or navies. A large but relatively worthless vessel would be loaded, in ancient times with dry wood and cloth, and in more recent eras with
gunpowder and other combustibles. A device indistinguishable from today's
molotov cocktail would be prepared, the intended fire ship launched in the direction of the assailing fleet, and the primitive bomb thrown onto its deck to hopefully break and explode brilliantly. Sometimes
sailors would steer the fire ship toward its target, use grappling hooks to attach the two ships, then escape and ignite the weapon. Unless the attacking fleet was trapped in a
tight spot, fire ships could usually be avoided easily, but they did have other uses, including destroying bridges and breaking up the formation of the enemy, sometimes drawing them into a trap.
More often than not, though, the use of fire ships either backfired or was ineffective, and the strategy tended to be deployed only in desperate situations. In 1587, Francis Drake was on a mission along the Portuguese and Spanish coasts to disrupt the planned Spanish Armada. In the Bay of Cadiz that spring, in Spanish-controlled Portugal, the garrison under his command set to destroying as many of Spain's ships as possible. After doing much damage on the first day of that expedition, the port's defenders on the second day set many of their own ships on fire in the hopes of doing some damage to Drake's fleet. No damage was done, and Drake's men are reported to have been loudly mirthful that the Spanish did their work for them.
In 1790, during a war between Russia and Sweden for control of the Baltic Sea, the Swedish navy released a fire ship against the Russians. However, the ocean currents brought it back into the midst of their own fleet, where it destroyed two other Swedish ships. The amused Russians presumably gained a much needed break from the pitched battle, which could be said to have ended in a tie.
Of course, fire ships did sometimes accomplish their aims, else the technique would not have survived for a millennium and a half. These aims were best accomplished when they were modest, however, as illustrated by the story of the pirate Charles Vane. In 1718, a new British governor was appointed for the Bahamas, and stamping out piracy was priority #1. He did this (temporarily), largely by granting amnesty, but Vane had not yet made his fortune and did not go along. He was chased by the British through a harbor, so set fire to a prize ship captured from the French, trapping the Brits in the habor's shallowest waters.
In addition to proper use of fire ships, I suspect that those which were set on fire with crew still on board were less likely to succeed, since sailors would probably be more concerned with saving their own hides than with accurate aim and such.