What you've heard is true: Japanese most probably does have the most difficult writing system in the world. Computer text entry has helped out a lot with that 'writing' bit -- if you can pronounce it, you can enter it into an IME and the program will do the hard work -- but reading Japanese remains massive linguistic hurdle.

Why Reading Japanese is Hard

A quick recap for those unfamiliar with the system: Japanese actually uses three different sets of signs for writing. The syllabaries hiragana ひらがな and katakana カタカナ are the easy part, there are around 50 of each and, with a very few easily learned exceptions, they never change their readings: ひ is always "hi" and カ is always "ka". It is entirely possible to read and write Japanese with these alone; that's the way Genji Monogatari was written 1000 years ago and that's how children's books are written to this day.

The clincher is the third set, the Chinese characters known as kanji 漢字。Basic literacy requires knowing about 2000 of these, and estimates of the number actually in use (including specialized fields) go as high as 5000.

So far, this is no worse than Chinese, which by most reckonings uses more distinct characters. However, while characters have one (or rarely two) readings in Chinese (esp. if we discount tone changes), in Japanese characters tend to have at least two completely separate readings. The worst offenders like 生 (sheng in Mandarin) have, depending on how you count, as many as 80 different ways to read them: JDIC lists

SEI SHÔ ikiru ikasu ikeru umareru umare umu ou haeru hayasu ki nama naru nasu musu u asa iki iku ike ubu umai e oi gyû kurumi gose sa jô sû so sô chiru naba niu nyû fu mi mô yoi ryû (phew!)
...with meanings ranging from "to give birth" to "draft beer"!

The reason things got so crazy is that Japanese and Chinese are entirely unrelated, Japanese just imported China's characters and then brute-forced them to fit Japanese. So-called onyomi (音読み, sound readings) are approximations of how they were pronounced in Chinese; kunyomi (訓読み), on the other hand, came about when existing Japanese spoken words were slapped onto Chinese characters. Compound words tend to use onyomi, while standalone words lean towards kunyomi, but there are plenty of exceptions. The process took over 1000 years and took place all over Japan with very little, if any, coordination, and it's a miracle the contraption works at all.

A Detailed Example

Consider Unicode 98DF:
This is a very common kanji (which is also a radical for building up more complex kanji); it is listed as number 289 (of some 2000) and is usually taught by the 2nd grade.
  1. The first stage of reading this is learning its meaning, in this case "eat" or just "food". See this somewhere, you know that food is involved.

  2. Next it would be useful to know the primary reading, so you can actually read it loud. In this case the most important form is the Japanese kunyomi reading taberu, a polite way to say "to eat".


    Get this far, and you can with some justification say that you know this character. But if you want to understand it thoroughly, there is still a long way to go.

  3. If the reading is a verb or adjective, and it often is, you also need to learn the okurigana, the trailing (lit. sending) kana used for conjugation. Taberu is written 食べる, so the character represents only the ta bit and the trailing beru has to be written in kana; in dictionaries this can be denoted as ta.beru.

  4. Once you've got that down pat, you need to learn other major readings. Most characters have both a Japanese kunyomi and a Chinese onyomi reading; in this case, the usual onyomi of 食 is SHOKU.

  5. But an onyomi alone won't get you far, you need to learn compounds featuring the character. Here examples would be chûshoku 昼食 "morning eat" (breakfast), shokudô 食堂 "food hall" (restaurant), shokuchûdoku 食中毒 "food poisoning", and less intuitive combinations like nisshoku 日食 "sun eat" (eclipse).

  6. Then we have secondary readings, such as kuu 食う, a vulgar term for eating; kurau 食らう, a more metaphorical verb for 'eating' prison sentences or bad weather; and the older onyomi reading JIKI 食, used in the odd word like kojiki 乞食 "ask eat" (beggar). Common characters, like this one, tend to have a whole slew of these.

  7. And finally there are the outright obscure readings, in very limited use and spelled out with furigana (helper kana) if actually used in print. The names of people and places (nanori), in particular, are notorious for completely off-the-wall readings; such as jii in 安食 Ajii (prob. corrupted from jiki), meshi in 食野 Meshino (shorthand for 飯野) and ke in 食満 Kema (completely random). But don't lose too much sleep over these, the Japanese have no clue about them either.

Why Reading Japanese Gets Easier As You Go Along

Fortunately, it's not all obscure readings and bizarre compounds.

The simpler characters, like 食, act as radicals (components, Jp. 部首 bushu) used to build up more complex characters. These often give some clue about what the complex character means (for example, nearly all characters featuring 食 involve eating or something eatable), and occasionally -- but not often -- you can even puzzle out the meaning, as in 餓 u.eru GA, "starvation", composed of 食 "eat" and 我 "self".

More usefully, Japanese words tend to be built out of multiple characters, and more often than not you can work out the meaning if you know the separate characters involved. 夕食 yûshoku "evening eat" must be dinner, 食欲 shokuyoku "eat desire" must be appetite, 人食い hitokui "human eat" must be cannibalism. This fails to work just enough to keep life interesting (浸食 shinshoku "soak eat" means erosion, not dehydrated food) and the reading is not always obvious (as in hitokui), but this is generally a great help and vocabulary builder. By contrast, English likes to import new, exotic and completely unrelated words, meaning that "dinner", "appetite" and "cannibalism" have to be learned by rote.

Last but not least, the further you work your way up in the character list, the meanings tend to become more precise and the number of readings becomes smaller. Past 1000 or so, most characters have (in practice) only a single meaning with a single onyomi reading. Once out of the Joyo Kanji 1945, most characters not only have a single meaning and reading, but are only found within a single compound as well: typical examples include the 醤 of 醤油 shôyu "soy sauce" and the 噌 of 味噌 miso "miso paste", which you will never, ever see used anywhere else.

So How Do I Learn to Read Japanese?

It's a recursive definition, but the only way to learn to read Japanese is to read Japanese. It's painfully slow going at first, decrypting the squiggles with a kanji dictionary in hand, but once you've slogged your way through your first book all on your own there will be no holding back.

Go for:

  • 3-panel cartoons in manga. Supremely easy to digest, although you'll miss many of the punchlines.
  • Books for children, preferably at about junior-high level; you want kanji (which elementary-level books don't have) and you want furigana on all kanji (which higher-level books don't have).
  • Books on topics that interest you. A friend's first book was a guide to parrot care, mine were a guide to Japanese pottery and Kneel and Lick My Feet, an Amy Yamada novel set in a Kabukicho S&M club.
  • Books by modern Japanese authors who write in simple, clear prose. Banana Yoshimoto and Haruki Murakami are excellent.
  • E-books (eg. 青空文庫 Aozora Bunko at http://www.aozora.gr.jp). No okurigana needed, just cut and paste, or use any of a number of "hover pointer over compound for a pop-up hint" programs. Alas, for copyright reasons the selection tends to favor older, tougher works.
Avoid:
  • Full-length manga. The language used is, generally speaking, pretty tough going, and often so are the plotlines. And you'll need to wean yourself from pretty pictures at some point anyway.
    • If you disagree, surf over to groups.google.com and check out sci.lang.japan/fj.life.in-japan, where the debate erupts into flamewars with depressing regularity.
  • Katakana-laden technical literature. My brain starts to fry after just a few pages of paketto from a burausaa kuraiento hitting a faiauooru instead of the saabaa because the konfigyureeshon is fakku-appu.
  • Any pre-WW2 literature. Yasunari Kawabata, Osamu Dazai, Natsume Soseki and (especially) Yukio Mishima may have won accolades, but their prose is far from what you're used to.
But above all... go for it! All you need is patience and perseverance, and the ability to read Japanese remains an highly unusual (unless you're Japanese) and valuable skill in any field.

References

Personal experience
JDIC