This has become naturally linked to British football in recent years, as we have seen a series of events involving violent clashes between supporters of rival teams and with the police. These riots have been seen as targetted both at property (local pubs, shops, etc. surrounding the football stadium) and at other people (police and/or rival team supporters).
Thanks to a BBC investigative programme, it was found that these confrontations were often organised and not spontaneous, as previously believed. As it turned out, groups of football hooligans were gathered together and organised by leader figures, who went out of their way to coordinate and transport the hooligans and instigate fights with the police or other football fans.
While it hasn't been shown that this is true for all football hooligans, there have certainly been cases recorded, and probably the general impression the public has is that football hooliganism is closely linked with the far-right, and right-wing political parties such as the British National Party and National Front (both regarded by mainstream political parties as fascist political parties trying to give racism a veneer of respectability). 'British' football hooliganism has also been linked more to English fans.
This problem has been so well-noticed in Britain that the police have made special efforts to work with foreign police forces in order to bar such hooligans from going abroad during the Euro 2004 competition being held in Portugal. This has been taken as far as suspects having their passports confiscated, to stop them leaving the country. This is similar to the banning of about 1000 football hooligans during the 2002 World Cup, stopping them from visitin Japan or South Korea (thanks to generic-man for this info.). Despite this, there have been reports of some suspects being able to bypass these measures and reach Spain or Portugal. It has been threatened that if British football fans 'misbehave', the England team may be expelled from the competition (something which, it seems, the football hooligans are less than concerned with).