Apparently, the first time the word "vandalism" was ever used was in a speech by Henri Grégoire, Bishop of Blois (1750-1831) to the National Convention of France. The Bishop used the word to describe the then-common act of burning public libraries.
In his Mémoires (1837, vol. 1, p. 345ff), he describes the circumstances:
"I created this word to end the matter {the library burnings} .... Curiously enough, this new term cause a literary controversy in Germany .... Respected scholars, born in the part of Germany where the Vandals had originated, claimed that the meaning I had conferred upon the word was an affront to their ancestors, who were warriors and not destroyers."