A modal particle is a grammatical particle found in some languages1 that indicates mood or intent, but does not fall into a clear syntactic category and are uninflected -- that is, they are not verbs, adverbs, etc., and they cannot be conjugated or otherwise modified through bound morphemes.
Perhaps the simplest example is the Japanese "ne", which is added to a sentence to indicate that you want confirmation2 -- in the same way that English uses "right" in the sentence "this goes here, right?"3
The German language is somewhat famous for having difficult model particles; this is in part because it is comparatively common for English speakers to learn German, as opposed to Japanese or Indonesian, but also because they are actually pretty complicated. For example, while Japanese modal particles usually fall at the end of sentences, German particles appear all over the place. German also has the annoying habit of using common words, such as ja, aber, and also as modal particles, and you have to rely on context to understand which usage is intended. And finally, in German these particles are informal, and may vary depending on dialect.
However, English is a lot like German, and the general use of model particles isn't too confusing if you don't overthink it; the German sentence Iss ja nicht das ganze Kuchen auf! translates fairly well as You had better not eat the entire cake!; the difference, important to linguists, that ja is a modal particle and had better is a modal verb phrase, is not important if you are simply learning the language. What is important is that these usages tend to be informal and idiomatic.
Because English does have a lot of modal constructs, some linguists use the wider term 'pragmatic marker' to refer to bits of grammar that indicate the type of message or the mood of the message, rather than give specific content.
1. Notably, German, Dutch, Japanese, Chinese, Indonesian and Russian.
2. This is an oversimplification, and this particle is used in other ways as well.
3. Why isn't the English 'right' considered a modal particle? Well, that's complicated, and I'm not entirely sure that I understand it, and I'm not entirely sure that anyone really understands it. There's no small debate over the existence of modal particles in English. However, in this case 'right' is a shortening of the phrase 'is that right', and thus is not a true particle.