I can't speak for people with other
first languages, but for English-speakers, German grammar does have some pretty difficult features to get around. Aside from some miscellaneous stuff, I would say that the difficulties fall into two major categories:
inflection and
word order.
Inflection
In grammatical (rather than phonological) terms, inflection is the process in which a word is changed to specify some further information about it, rather than to alter its basic meaning. Some languages use this extensively for all sorts of words, while in others it barely features at all. Latin, Finnish and Old English are examples of highly inflected languages; Chinese is at the opposite end of the spectrum. English lies strongly toward the uninflected end, while German by comparison is highly inflected.
An example of inflection in English is this. If you were talking about yourself, you may say "I got up this morning," or you may say "My sister woke me up this morning." The words "I" and "me" refer to exactly the same person, yet they are clearly different - the former specifies that you were the subject of the verb (the doer), while the latter shows that you were the object (the verb was done to you).
A more obscure example is the distinction between "blond" and "blonde" in English. One is used for males, the other for females - the use or absence of the letter "e" does not change the meaning of the word, but rather makes it more specific.
Inflection in English, which was once extensive, is now used almost exclusively for pronouns, as in the distinction between "I" and "me", or "he" and "him". German, however, still uses it for verbs, nouns and adjectives to indicate gender, case, number, and a fourth property of adjectives so hopelessly obscure and pointless I'm not even sure if it has a name.
To take these four in turn:
Gender is a property of nouns found in almost all European languages, but not English (except for pronouns, i.e. he/she). Nouns in German can be either masculine, feminine or neuter. To indicate this distinction, you must use the correct definite/indefinite article, and the correct ending for any adjectives preceding the noun. Thus "an old man" is "ein alter Mann", but "an old woman" is "eine alte Frau". This concept is alien to the English-speaker.
Case is a property of nouns found in all languages, but to a greatly varying degree. The case of a noun indicates its role in a sentence - whether it is doing a verb, having a verb done to it, possessing something, or whatever. The distinction between "I" and "me" in English is a case distinction, as is the use of "'s" at the end of words to indicate ownership, e.g. "Barry's house".
German nouns have four cases - nominative, accusative, dative and genitive, for the subject, object, indirect object and possessor respectively. To indicate case, you must always use the correct articles and adjective endings, as with gender. For example, "he is an old man" is "er ist ein alter Mann", but "I have seen an old man" is "ich habe einen alten Mann gesehen". "Ein alter" has transformed magically into "einen alten", because the man is now having a verb done to him. Get this wrong and you will look stupid.
Case always applies to articles and adjective endings, but for a certain class of masculine nouns, known as the weak nouns, it also alters the noun itself. Take "Polizist", meaning "policeman". "I am a policeman" is "ich bin Polizist". But "I know a policeman" is "ich kenne einen Polizisten". "Ein" has changed to "einen", but "Polizist" has also gained an -en and become "Polizisten", because it is having something done to it - it is in the accusative case.
Additionally, all plural nouns in the dative case must end with an "n", which for most of them means they change. "Männer", for "men", becomes "Männern" when in the dative case. The exception is those nouns, usually foreign imports, which add an "s" in their regular plural - "Radio" is an example.
Number indicates whether a noun is singular or plural. In English this is done by adding "s" to nouns to make plurals. German also makes plurals by adding a word ending, although there are several of these, including "e", "n", "en" and "s". What is unusual about German, from an English-speaking point of view, is that it also inflects verbs and adjectives according to number.
In English, conjugating a verb is child's play - "I sing", "You sing", "He sings", "We sing", "They sing". Apart from the added "s" to the third person singular, the verb form - sing - stays the same. In German, plural verbs are distinguished by adding a "t" or "en" to the verb stem. "I sing" is "ich singe", but "we sing" is "wir singen", "they sing" is "sie singen", and you (plural) sing is "ihr singt".
Number also comes into play for adjectives. "An old man" is "ein alter Mann", as we've seen, but "old men" is "alte Männer". Because the long-suffering old man has brought some friends along, the adjective changes. As far as adjective endings are concerned, plurals effectively behave like a fourth gender.
One final aspect of inflection, which really makes you understand why the Germans got so pissed off around 1914, has to do with the words preceding an adjective. Depending on whether an adjective is preceded by a definite article, an indefinite article, or no article, its ending will change. These three different types of inflection are called "strong", "mixed" and "weak".
"The big house" is "das große Haus". But "a big house" is "ein großes Haus". An "s" has been added for no reason discernible to mankind. (Actually there is a reason, but it's not a very good one.) Meanwhile, standing impatiently outside the house, are our friends the old men. "The old men" is "die alten Männer". But "old men", without a preceding "the", is "alte Männer". At the risk of editorializing, this is a pointless waste of mental energy.
Word order
This is the other big thing you have to worry about. There are some minor aspects of idiom that differ from English - for example, in German you say "I'm going next week home", rather than "I'm going home next week". This is to be expected in any language. There are however some standardized differences that are prone to confuse the English-speaker.
The first thing you learn is that infinitives and past participles must go to the end of the clause. In German you say "I have a new house bought", or "next week I will a new house buy". This can take some getting used to.
Additionally, there are certain clauses in which the finite verb must also go to the end. Consider the English clauses: "The man who owns the house"; "He told me that it might work"; "Because he works there". In the German translations, the finite verbs - the words "owns", "might" and "works" - would go to the end of the clause. Thus in German, you do not say "because I have bought a new house", but "because I a new house bought have". This is perhaps the silliest aspect of all German grammar.
The abundance of case-indicators in German also means that the key nouns and verbs in a sentence can be moved around with more freedom than in English. The example I learnt in school was this: to say "the dog bit the postman", you can use the same word order as in English - "der Hund biss den Briefträger". But because "der" and "den" are case-specific, you can also choose to say "den Briefträger biss der Hund". The meaning remains the same, because the definite articles show which noun is the subject and which the object. It can't be done in English, because "the" expresses nothing about case; but to the Germans it is second nature. This is alien to the English-speaker, but it can also be useful, particularly for creative or expressive writing.
There are various other aspects of the German language that can also leave the foreigner high and dry, but these for the most part come under idiom rather than grammar.
Many of these are common to most European languages, and English is often the exception. One example is the use of two levels of formality - "du", "dich", "dir", "ihr" and "euch" for the informal second person, and "Sie" and "Ihnen" for more formal address. Another is the tendency to drop the indefinite article - a German would say "I am doctor", not "I am a doctor". A third is the use of "be" as an auxillary, rather than "have" - for example, a German (or Italian or Frenchman) would say "I am never been there", not "I have never been there".