When the Americans invaded Iraq, they were invading one of the most tightly-controlled societies in the modern world. The famous Iraqi dissident Kanan Makiya called it a "republic of fear", implying that fear was the emotion that dominated the Iraqi res publica, the public thing. But it was not a democracy of fear; although ultimately no-one was safe, some had more to fear than others. The country's Sunni minority - at least those trusted by Saddam Hussein - sat at the country's helm. They, quite literally, knew where the bodies were buried. When the Americans came, it was them who had the most to lose.

One feature of the faulty worldview which characterized the run-up to the Iraq invasion was the belief among some Bush administration officials that they were essentially reliving the war against Nazism. The way they saw it, a vicious ideology - Ba'athism, which in their defence does at least bear some resemblance to Nazism - had cruelly gripped a nation, and if they could merely eliminate the regime which enforced this ideology then there would be singing in the streets and a wondrous post-sectarian future for all. "Normality" would return, which would mean democracy and economic growth posited on free markets. The Bush administration split Iraqis into two groups - those associated with the regime (bad) and those not associated with the regime (good), and saw the key to the country's future in disempowering the former and allowing the invisible hand of "normality" to run its course.

This proved to be an incredibly faulty set of assumptions on which to stake the future of a nation.

Several practical consequences can be said to have derived from this worldview. Firstly, the American military did not focus on the physical liquidation of the regime's power structures; many of the police and army personnel, it was reasoned, were coerced into serving and would melt away once Baghdad had fallen. Instead, the U.S. placed a lot of store in the institutional liquidation of the regime's power structures - they disbanded the army and the national police, and banned many people who had been members of the Ba'ath party from serving in the civil service. These people were mostly Sunnis, the people who had the most to fear in an Iraq dominated by the Shiites who they had oppressed for so many decades. The tension between the Sunnis - who feared an oppressive Shia dictatorship - and the Shia - who feared the re-emergence of an oppressive Sunni dictatorship - was to prove the key issue in Iraq's development.

The Sunnis went home with their weapons and they waited to see what happened next. The U.S. never made any serious attempt to disarm them, and there weren't enough U.S. forces to secure all or even most of the major population centres in Iraq anyway. Uncle Sam took special care to stay clear of the Sunni areas in Anbar province, western Baghdad, and north of Baghdad. This "Sunni triangle" became the nexus of Iraq's insurgency, where literally hundreds of thousands of men who held arms under Saddam's regime peered at the machinations of the Americans and their Shiite allies with increasing suspicion and fear. Suddenly stripped of the glories of power, feeling disenfranchised in the present and terrified of the future, they still had one thing left: their guns and their bombs. They knew where the arms caches were buried, as well as the bodies.

Soon, they were putting this knowledge to gruesome use.

* * *

The Iraqi insurgency is much more complicated than most cursory media analyses would lead one to suspect. From the beginning, the distinction and conflict between Sunni armed groups and Shiite ones has been important, and from about 2005 onwards the relations between the various armed groups became much more important than their position vis-a-vis the occupier. Iraqis are not stupid and can watch satellite television - at least, when religious extremists are not stopping them. They know that the U.S. has a very limited stomach for foreign adventures, and that eventually the occupiers are going home. They have seen Black Hawk Down. They did not commit the bulk of their resources to attacking the Americans, but set out to be in the best position to win power once the Americans had finally left. Getting yourself killed by a U.S. soldier before this denouement would have been a needless waste.1

And U.S. military policy only encouraged the Iraqis in holding this view. Until 2007, the dominant school of thought in the U.S. military and among policymakers in Washington was that the U.S. military presence was actively provoking the insurgency, and the key to ending it was to keep visibility as low as possible, train Iraqi troops and police to provide security, and leave as soon as possible. Military units kept to themselves on huge and impregnable bases - remember the image of the Green Zone during those years, so isolated from the Hell outside - and Iraq descended into a Hobbesian nightmare as Sunni and Shia vied to get on top. Baghdad's civil war began to rage in full thunder, and most of those civilians who died went to their graves not because the Americans killed them, but because the Americans failed to stop their own countrymen from killing them.

But behind the endless drumbeat of violence, complicated sectarian and tribal politics were at play. The basic contours of the Baghdad civil war can be understood like this: Shia eastern Baghdad, led by the armed group known as the Mehdi Army, was slowly taking over western Baghdad by pushing the Sunnis out into the desert, and the Sunnis were responding with car bombings and mass killings in Shia areas. But the Shia were winning, especially because the Iraqi government was sympathetic to them and the army and police was riddled with their supporters or active militia members. A Sunni was more likely to be imprisoned and tortured in the Ministry of Health that he was to receive basic health care. Electricity hummed for 18 hours a day in the east, but the Sunnis areas were lucky to get two.

Faced with such desperation, the Sunni population turned to al-Qaeda in Iraq, the most extremist of the Sunni groups which brought with it a brand of puritanical Islamism which was alien to Iraq's tradition. This was a big step for most of the other Sunni armed groups - the main two are called the 1920 Revolution Brigades (after a famous nationalist uprising) and the Islamic Army - because Saddam's regime had been strictly secular, and most of the Sunni insurgents were former participants in that regime. But al-Qaeda brought with them huge amounts of money (from the usual places one can rely upon to fund armed Islamic extremism - private donors in Pakistan and Saudi Arabia, and the government of Iran), and they brought an ideology which was much easier to sell than Ba'athist revivalism - at least at first. They also brought foreign connections and expertise, something that it seemed would be needed against the crushing domestic might of the Shiites.

But the protection of al-Qaeda in Iraq came at a price. They quickly came to dominate many major Iraqi cities which had majority Sunni populations - Fallujah, Ramadi, Samarra, Mosul, Baquba, and western Baghdad. And where they dominated, they imposed their harsh brand of Islamic law - the shar'iah had not been seen in Iraq for a long time, and it was not particularly welcome now. They cut off hands, banned alcohol and foreign products, and made it mandatory to wear a beard; the Iraqi government and the Americans were powerless to stop such deviations from the country's law, and tiny emirates sprung up in emulation of the Taliban wherever one looked in the Sunni triangle.

Meanwhile, the tectonic plates of power shifted below the surface, and suddenly they caused an eruption. This was the birth of the Sons of Iraq.

* * *

The traditional centres of power in Sunni Iraq are tribes. Tribes - extended family groupings - look out for each other and have loyalties which cannot be necessarily severed by institutional affiliation, say to the state or to an insurgent group. Tribal sheikhs have the power of life and death in Anbar province, and along with all the other associated irritants, al-Qaeda was particularly painful for them because it meant a diminution of their traditional power. Because Sunnis had widely boycotted the 2005 elections in Iraq, they also found themselves woefully under-represented on the political scene and hence without influence. Squeezed between a hostile Shia government and Sunni fundamentalists, the sheikhs realized they had one ally left who might help them: the U.S. military.

It started in Anbar province in 2005. U.S. forces heard the nightmare of gunfire only to perform reconnaissance and discover that the battle was, in military parlance, "red on red" - insurgents fighting insurgents. Soon, the local sheikhs approached U.S. forces and proposed an alliance: they would stop attacking the coalition if the coalition would help them tackle al-Qaeda. For the U.S. military, it was a win-win deal - not only would the tribal groups no longer be firing at the coalition, but they would also be helping them to track down al-Qaeda. Meanwhile, the sheikhs got the opportunity to demonstrate that they could participate in a constructive fashion in Iraq's future; this seemed their best hope of eventually achieving reconciliation and a modus vivendi with the Shia government in Baghdad.

The movement - which eventually spread across Sunni Iraq - was originally called "the Anbar Awakening" by the Iraqi participants. U.S. forces originally referred to the men involved as "Concerned Local Citizens" but later settled on "Sons of Iraq" because apparently the former was unwieldy when translated into Arabic. The supreme irony was that most of the men involved in the programme had been insurgents and hence until recently labelled with the term "Anti-Iraq Forces", which was the name the military used to favour for insurgents. The journey of these concerned citizens from enemies of Iraq to its progeny marked a cultural shift in the way the U.S. fought the war in Iraq. Suddenly there was a new focus on the complexities of the Iraqi civil war, and how they could - indeed, must be - harnessed to bring stability to the country.

The Awakening Councils brought vast benefits to the U.S. and Iraqi governments. Firstly, because they were former insurgents and were embedded in their communities, they had intimate knowledge of al-Qaeda's operations. Gradually, the Awakening Councils helped U.S. counterterrorism forces pick apart al-Qaeda, removing much of the rationale for the existence of the extremist Shia militias, who claimed they were defending their neighbourhoods against Sunni terrorists. They were also enlisted to provide security once al-Qaeda had been driven out of an area. They knew who lived in a certain village and who was an outside infiltrator; they knew where everyone's sympathies lay; and they were trusted by the local population to provide security. Iraqis overwhelmingly prefer to have locals provide security and policing, especially when outsiders could come from a different sectarian group. And the Sons of Iraq now benefitted from the active help of the U.S. military in keeping their neighbourhoods safe from both al-Qaeda and Shia extremists.

But the Maliki government remained highly suspicious of the Sons of Iraq. Many of them were former insurgents with blood on their hands, and while the U.S. military proved itself institutionally capable of overcoming the past, the Shia security forces have not done. They worried that the Americans were building up an effective armed organization which could be used to restart an insurgency, and they contended that the Sons were just opportunists who took U.S. money while they could and would return to the insurgency when they had to. The Sons of Iraq were indeed paid salaries by the U.S. after switching sides, but it is hard to ascribe their actions entirely to economic motives since al-Qaeda are known to pay more. The substantial price paid in lives by the Sons in their battle against extremists can likewise not be explained by $30 a month.

The Sons of Iraq turned to the coalition largely through desperation, but also because they pondered the ever-present question in Iraqi politics: what is going to happen to you when the Americans leave? If al-Qaeda remained in their midst, it would not only make their life intolerable but also give the Shia government an excuse to oppress them and malign their whole community. When the Americans left, the Shia would have the power, and the Sunnis would have to live with it. Something had to be done to change the status quo before this eventuality came about, and the opportunity came in the form of America's interest in reconciling moderate Sunnis and promoting a more even distribution of power in Iraq. It was a masterful game of realpolitik being played by the tribal sheikhs, but it was also a risky one.

* * *

In late 2008, the U.S. military handed over control of the Sons of Iraq programme to the Shia government as part of the restoration of full Iraqi sovereignty. More than anything else, how the transition is handled will determine the future of Iraq. The Sons comprise 103,000 armed men with a past in the insurgency and a possible future in it; collectively, they have the power to reignite the country's civil war. The government remains suspicious of them and sees them as more of a security risk than a boon; it has promised to integrate them into the national police and army, but progress on this has so far been slow. Meanwhile, the Sunnis fear that the government aims to disarm the Sons and launch new assaults on the rights of the Sunni community - they are hence unwilling to put down the weapons which they see as the only sure guarantor of their future. Delays by Baghdad in paying the salaries of the Sons, as well as the disbanding of a prominent Awakening group that was apparently still balls-deep in the insurgency, has only heightened Sunni fears.

With U.S. forces now withdrawing, the future looks even more precarious for the Sons of Iraq. Spokesmen for some of the groups have said they have been "betrayed" by the U.S. military, who promised to protect them. But the U.S. cannot protect them for ever, and the future of the Sons is a key test of the Iraqis' ability to live with one another without the provision of security by the United States. As the American military footprint shrinks and retreats back to large bases beyond city limits, the U.S. force posture will begin to look the same as it did in 2005, when the civil war began in earnest. This time, to be sure, things are different - the Iraqi military is vastly more competent and capable of coping on its own, al-Qaeda is in a strategic retreat from which it would be incredibly difficult to recover, and the horrific memories of the last few years warn against renewed conflict. But what is not different is the pervasive sense of fear.

Later this year, Iraq will hold new national elections which will lead to the formation of a new government in Baghdad. Unlike in 2005, Sunni participation is expected to be widespread and the new government is expected to be of a less sectarian bent. U.S. forces remain in place until then to prevent Maliki taking steps to reduce the fairness of the elections or to stoke sectarian passions before them. Iraq's politics is so young and unpredictable that it would be foolish to try and forecast the result, or predict how it will affect the security situation. But one thing that can be said for sure is that if Iraqis do not build on the courage and realism of the Sons of Iraq and instead dwell on their long-nurtured fear and loathing, they will never find peace together. Time is running out.

1. Consider insurgent tactics in Iraq as opposed to, say, Afghanistan. Stories of insurgents attacking coalition strongpoints in company or even platoon sized units are rare in Iraq - and mostly occurred when the insurgents were desperate during the surge - whereas anonymous bombings and mortarings are the norm. This was not because Iraqi insurgents lacked proficiency in unit combat tactics - many were ex-soldiers - but because the expenditure of their resources in this fashion was not commensurate with their goals.

You can read more about the Sons of Iraq in books like Linda Robinson, Tell Me How This Ends: General David Petraeus and the Search for a Way Out of Iraq, Bob Woodward, The War Within: A Secret White House History, and David Ricks, The Gamble: General Petraeus and the Untold Story of the American Surge in Iraq, 2006 - 2008.

Also on the war in Iraq:

Intelligence and the 2003 Iraq War
Our tomorrow starts today: one minute to midnight in Iraq
Al-Askari mosque
Iraqi-U.S. status of forces agreement
Refusing to run: the U.S. troop surge in Iraq