Although a katana is indeed worn thrust through the belt, on the left hand side with the edge up, a katana is not the only kind of Japanese sword.
A tachi, for example, is worn in a sling with the edge down.
This is mostly a matter of tradition, but it is easy to see that a curved slung sword will be more stable edge down. Similarly, some experimentation will show that a curved sword stuck in your belt will be more secure with the edge up.
When tachi were commonly worn, it was common to have a short katana as a backup blade -- not unlike the way a wakizashi is worn with a katana. Since the katana was worn as a secondary weapon, it was kept thrust through the belt.
At about the same time that heavy armor, like oyoroi, was replaced with lighter and closer fitting armor, like haramaki and domaru, it became common to wear a single, longer, katana instead of the tachi. Even after that point, generals still wore full oyoroi with slung tachi on the battlefield.