David Petraeus is a spry and youthful 55-year-old, of medium height and build. Perhaps because of two accidents in his army career – he was shot in the chest during an exercise and smashed his pelvis in a parachute jump – he has a slight stoop and a barely perceptible air of physical awkwardness.

Instead of the ramrod bearing of a MacArthur, Gen Petraeus resembles a modest headmaster, albeit one with a personal fitness obsession.

This outgoing yet scholarly figure is now the brightest star of America’s armed forces. Gen Petraeus has emerged from the bloodshed of Baghdad to become the only genuinely successful general of the “war on terrorism”. No other commander in this campaign, which has lasted exactly seven years, would have reached the cover of Time magazine as “Man of the Year” in 2007. If every war eventually makes a military reputation and propels a uniformed figure to global fame, then Gen Petraeus is the Eisenhower of our time.

When he hands over his command in Iraq on Tuesday, he will have achieved a seemingly impossible goal. Gen Petraeus will leave the country in infinitely better shape than found it. The West’s enemies – from “al-Qaeda in Iraq” to the Shia gunmen of Moqtada al-Sadr – are greatly weakened.

Is this achievement genuine – or was Gen Petraeus simply lucky enough to take over at the right time? And if his accomplishment is real, how has he done it?

Luck has smiled on Gen Petraeus to the extent that he has always seemed to be in the right place to learn the lessons of earlier failures. He graduated from West Point in 1970, just in time to imbibe the hard messages of Vietnam without being personally scarred by the war. This led him into a detailed study of counter-insurgency warfare, a subject that became his consuming interest. In 1987, he went to Princeton and produced a 328-page thesis on the impact of Vietnam on America’s high command. Fast forward to March 2003 and the Anglo-American invasion of Iraq. Gen Petraeus was in command of a spearhead unit – the 101st Airborne Division – and found himself in control of the Shia holy city of Najaf. He had no particular wish to run the place, but amid the breakdown of law and order unleashed by the war, he could not find a civilian mayor to take over. He was in the right place to grasp the central mistake of his high command: the invasion had decapitated Iraq’s leadership without deploying enough troops to secure the country and prevent its spiral into chaos.

Gen Petraeus completed another tour of duty in Iraq in 2004/5, when he was in charge of training the new army and police force. These units were America’s “exit ticket” from Iraq, in line with President Bush’s statement that “we will stand down as the Iraqis stand up”. But Gen Petraeus was later criticised for over-optimism and for greatly overestimating the abilities of the newly trained formations. Many were infiltrated by militias and proved unreliable in battle.

Once again, he was in an ideal position to learn some hard lessons: declarations of victory must be avoided and American soldiers would not be able to hand over to their Iraqi counterparts in the near future. Instead, they would have to stay for longer than he once thought.

After this tour, Gen Petraeus returned to America and wrote what was to become the book on counter-insurgency warfare. Together with a Marine Corps general, he produced the US military’s first field manual on this vital subject for 20 years. The 282 pages are a tersely written, closely argued set of instructions on how to win supposedly unwinnable wars. Drawing on a host of historical lessons – from Vietnam to Lawrence of Arabia’s campaign against the Turks – the book could be read as a manual on how to rescue the situation in Iraq.

The central insight was that protecting civilians was the sine qua non for beating insurgents. If people felt safer, they would back the security forces and turn against the gunmen. But this required far more troops than were in Iraq at that time. Most importantly, it also required those forces to live among the population and fight in a completely different way.

Once again, Gen Petraeus was in the right place at the right time. Just as he became the army’s acknowledged expert on counter-insurgency warfare – and the author of an unofficial manifesto for winning in Iraq – President Bush was casting around for a new strategy. After the bombing of a Shia shrine in Samara in February 2006, the bloodshed had escalated to become a sectarian civil war. By the summer, a wave of attacks was killing 2,500 civilians a month.

Mr Bush lost confidence in his military leaders and ordered his civilian advisers to launch a comprehensive policy review. They came up with two key recommendations: send a “surge” of another 30,000 troops to Iraq and adopt the new doctrine of counter-insurgency warfare.

America’s service chiefs were unanimously opposed to the “surge”. They feared the strain on the military machine and gave warning that too few troops would be left to cope with emergencies elsewhere. But in January last year, Mr Bush overrode his key generals and announced that another five army brigades and 4,000 Marines – almost 30,000 troops in total – would go to Iraq. The new units would be concentrated in Baghdad, where force levels would rise from 17,000 to 40,000.

A new strategy required a new commander, and Gen Petraeus was the obvious man.