The long road leading to No 10

The new coalition may have come as a shock to many, but the courtship has been going on for years. Political Editor Jane Merrick reports

Britain's new Prime Minister David Cameron speaks with his new Deputy Prime Minister Nick Clegg, as they pose for pictures outside 10 Downing Street last week (GETTY IMAGES)

In late January 2006, when the Liberal Democrats were at a dangerous crossroads after the resignation of Charles Kennedy and sexual shenanigans of leadership candidates, a long-term friend of new Conservative leader David Cameron gleefully predicted the third party of British politics could split in half – with the brightest stars defecting to the Conservatives.

In the Ways and Means corridor that leads to the members' lobby of the Commons, the MP said: "We think David Laws is a brilliant talent. He would make an excellent Chief Secretary to the Treasury under Prime Minister David Cameron."

When, later that day, Laws was told of the remark, he said: "This is drivel of the first order and a cack-handed attempt by the Conservatives to try to destabilise the party."

More than four years on, Laws is indeed Chief Secretary to the Treasury under Prime Minister Cameron. The only part of the prediction that was wrong, of course, was that the MP for Yeovil did not defect but is an enthusiastic new member of the Liberal-Conservative coalition.

A Lib Dem-Labour pact, with a new Labour leader replacing Gordon Brown, was described as the "shotgun wedding" scenario, but the Cameron-Clegg coalition is a union that has been several years in the courtship. This courtship did not involve formal defection talks – although George Osborne once tried to get Laws to cross the floor – but an organic convergence of ideology between the modernising wings of the Conservatives and Lib Dems, motivated by a common desire to make their parties electable.

This sheds light on why, while it came as a shock to some Conservative and Lib Dem activists, the Cameron-Clegg deal struck last Tuesday was more natural and instinctive than a Lib Dem-Labour coalition. During five days of negotiations last week, there were a number of factors that killed off chances of a Lib-Lab pact and led to the Lib-Con deal being struck, but the favourable conditions for the historic partnership were created nearly a decade ago by two groups of modernisers in each party.

For the Tories, from early in the last decade, it was those starting to gather around two young MPs, David Cameron and George Osborne, who saw the pair as the brightest hopes to lead the party into government.

For the Lib Dems, it was the economic liberal tendency, which included the former banker David Laws and two pragmatic MEPs, Nick Clegg and Chris Huhne. In 2004, they contributed to The Orange Book, a collection of essays which stimulated the realignment of the Lib Dems that has culminated in their entry into the Cabinet for the first time in decades.

Between 2002 and 2005, a member of each of these tribes found themselves sitting at desks opposite each other while working as political advisers for former Lib Dem leader Lord Ashdown, the international community's special representative to Bosnia.

One was Ed Llewellyn, now Cameron's chief of staff in Downing Street; the other was Julian Astle, who would later run the CentreForum think tank, the spiritual heart of modernising Lib Dems. When they were not helping Lord Ashdown, the two young advisers would sit in the office in Sarajevo debating the future of their respective parties. Both faced a similar problem: the Lib Dems under Charles Kennedy and the Tories under Iain Duncan Smith and later Michael Howard were struggling in the polls and looked certain to lose the 2005 election. Both had clear thoughts about the direction their own party should be going.

In 2004, Astle received a package in the diplomatic bag from the UK: Laws, the Cambridge double first graduate and a former JP Morgan vice president, had sent him a copy of The Orange Book he had co-authored. The book called for a radical repositioning of the Lib Dems based on free-market economic liberalism and condemned "soggy socialism". Astle was gripped, and read out excerpts to Llewellyn, who listened intently.

One of The Orange Book essays was written by Clegg, the MEP and parliamentary candidate for Sheffield Hallam for the 2005 election. In the 1990s, he had worked for Leon Brittan, then European commissioner. The former Tory cabinet minister told colleagues in Brussels his protégé was "the best political prospect of his generation". While he has since pushed his pro-European credentials, Clegg's Orange Book essay argued for a repatriation of powers from Brussels: ironically, one of the demands of the Tory right for the Cameron government, and one that has been watered down in the new coalition's agreement.

Laws, in a chapter entitled "Reclaiming Liberalism", raged: "How did it come about that over the decades up to the 1980s the Liberal belief in economic liberalism was progressively eroded by forms of soggy socialism and corporatism, which have too often been falsely perceived as a necessary corollary of social liberalism?"

The Tory modernisers coalescing around the Cameron project faced the opposite problem: to move the party from the right to the centre ground. A Lib Dem on the right of the party said: "In the Conservative Party, there were people around Cameron who understood what had to be done to make themselves electable, while in the Lib Dems there were those who wanted the party to be more ambitious, more professional, to make that transition from a party of opposition to a party of government. A critical mass of people in both parties began to emerge."

Within a year, as Ashdown's tenure in Bosnia was coming to an end, Llewellyn had quit to work on Cameron's leadership campaign, while Astle returned to Britain to set up the CentreForum think tank.

The Lib Dems' 2005 conference was disastrous for the leader Charles Kennedy, with MPs openly questioning his rule. But on the sidelines, an intellectual tug of war was being fought. Clegg, by now an MP and spokesman on Europe, spoke alongside Vince Cable, the party's Treasury spokesman, for a motion to freeze the EU's budget. When it was defeated by activists, Cable condemned those opposed to party reform as "ostriches".

A second right-leaning motion – to privatise the Royal Mail – was also defeated. Norman Lamb, another Lib Dem MP sympathetic to the Orange Book tendency, came close to tears after the vote. Lord Greaves, from the left of the party, angrily confronted Lamb, declaring: "The economic liberal people have got to realise they are going to lose the debate."

But the Lib Dem peer was wrong. Cameron was elected leader in December 2005. Within weeks, Kennedy resigned after an open revolt at the top of his party, precipitated by a sudden realisation that the new Tory leader would gobble up the centre ground from the right, and squeeze the Lib Dem vote. The reason for his ousting was more than that he was an alcoholic.

The Lib Dem 2005 election campaign had been boosted by the party's opposition to the Iraq war, but the Orange Book tendency was frustrated that the manifesto contained "intellectually lazy tax and spend politics", to quote one of its members.

Those close to Cameron, appraised of the ideological battle raging inside the Lib Dems, saw an opportunity to woo those from the right. The Tories could make up ground in an election by taking seats from the Lib Dems in the South-west. A courtship of Lib Dems began, but for some close to Cameron the aim was less about unifying the two parties than creaming off the best talent from the right, which would lead to the destruction of the party at its activist base.

The Cameron ally who spotted Laws's talent as a potential chief secretary to the Treasury also identified Cable (now Cameron's Business Secretary) and Jeremy Browne (now a Foreign Office minister in the Lib-Con government) as would-be targets.

The 2006 leadership contest, initially dominated by the sex lives of candidates Mark Oaten and Simon Hughes, eventually saw Sir Menzies Campbell beat Chris Huhne, the only one of the "Orange Bookers" to run. Sir Menzies did professionalise the Lib Dem operation, but in the eyes of modernisers, the elder statesman was not going "fast enough and hard enough" on reforms, allowing Cameron's party to capitalise on what he called the "liberal conservative" agenda.

In March 2006, shortly after Sir Menzies' election as leader, Ken Clarke, as a Tory moderate, was sanctioned by Cameron to court openly Lib Dems. Clarke was vexed – in the end, rightly – that the next election would lead to the Tories winning more votes than Labour but falling short of an outright majority on seats. Clarke told The Spectator that month: "If we're the biggest single party, then we must turn to the Liberals and try to persuade them to join us. I don't think we'd find it impossible." He added, prophetically: "I'm glad to say the fates could condemn the Conservatives and the Liberals to form a coalition."

Later that spring, Laws was directly approached by George Osborne asking him to consider defecting to the Conservatives. Laws later joked: "One or two of my colleagues may have thought I already was."

Sir Menzies eventually stood down in autumn 2007, and Clegg narrowly beat Huhne as his successor. In his first conference speech as leader, in March 2008, Clegg laid out a tax-cutting agenda, where any spare public money would be used to reduce the bills of families rather than be spent on public services. He said he was interested in building a "new type of government" that was "based on pluralism instead of one-party rule".

Clegg's allies insisted that the party remained "equidistant" between Labour and Conservatives, and in public, he attacked David Cameron and Gordon Brown with equal force. But in private, Westminster-educated Clegg found himself more at ease with Cameron than with Gordon Brown. One notorious incident was the cross-party talks on expenses last summer, when Clegg was aghast at being "basically ignored" by the Prime Minister.

And in October last year, at the official opening ceremony of the Supreme Court, Cameron and Clegg found themselves alone in a room together as they waited for proceedings to start. Neither was in the royal party nor the Prime Minister's party, so it was an opportunity to spend time together. The body language was "warm" – similar to that on display in the Downing Street Rose Garden last week.

The Lib Dem leader maintained the public "equidistant" line until November last year, when he told the BBC that whichever party had the most seats in a hung parliament had the "first right" to seek to govern.

When it became clear on the morning of Friday 7 May that the outcome was a hung parliament, Llewellyn – a personal friend of Clegg's wife, Miriam – made contact by telephone with the Lib Dem team.

Clegg publicly announced he would talk to the Tories first. Cameron then made his "comprehensive offer" to Clegg, which would include an inquiry into electoral reform. Over the weekend, the two parties' negotiating teams got to work hammering out a deal. On Saturday evening, Cameron and Clegg met alone at Admiralty House. It was all coming together.

But Clegg and his team were not yet ready to cave in without exploring the option of a Lib-Lab pact. So the Lib Dem negotiators talked to Labour, and Clegg spoke by phone to Brown over the weekend.

Last Monday morning, formal talks between the Lib Dem and Tory teams began at the Cabinet Office, with the press camped outside. Away from the cameras, other activity was going on. Lord Ashdown opened up a channel to David Miliband, the frontrunner to succeed Gordon Brown. The former spy suggested to Miliband that the Lib Dems would run into the arms of Labour if the Foreign Secretary would make clear he was ready to take over from Brown. This was the "shotgun wedding" scenario that was first suggested by Labour figures as long ago as last December.

Miliband rejected the offer. Ashdown also blamed the "Neanderthal knuckle draggers" in the Labour Party – John Prescott, David Blunkett and John Reid – who publicly declared that a Lib-Lab coalition would not work.

By Monday afternoon, the Tories – alarmed to learn that talks were active between Labour and Lib Dems – offered a referendum on the alternative vote system. It was the minimum giveaway on the spectrum of electoral reform, but it convinced enough senior Lib Dems that the Tories were serious. Labour's negotiators went further than the Tories on electoral reform – even offering a referendum on the single transferable vote system, the one the Lib Dems wanted. More compelling was that together the Lib Dems and Tories made a majority.

In the end, just as it was Lord Ashdown who brought together Lib Dem and Tory modernisers in his Sarajevo office nearly a decade ago, it was the peer who would be key to the Lib-Con coalition deal being struck.

By Tuesday, as talks were still ongoing between all sides, Ashdown was shown a copy of the draft coalition agreement, revealing the Tories had made considerable concessions on tax, Europe and electoral reform.

Despite his personal friendship with leading figures in the Labour Party, Ashdown let it be known to Clegg that it was a deal that could be done – and later that day it was.

A senior Lib Dem figure said the deal could never have happened without the years of quiet, behind-the-scenes courtship and ideological convergence. Another likened the two parties to two towers that stand side by side but are closer together at the top – with Cameron and Clegg nearly inseparable – yet at the bottom, they are further apart. This does not bode well for stability of the pact.

One Lib Dem insider has a gloomier assessment: the Conservatives have attained power by subsuming senior Lib Dems into government and making concessions on policy to show they are truly "detoxified", with little care for the third party's electoral support. There are grim warnings that next year's local elections will see Lib Dem votes plummet.

The insider compares the Tory party to a praying mantis that mates with a partner before eating it. "Being charmed by David Cameron and his friends is like being bathed in warm honey. You have to be careful not to drop your defences. Unfortunately, it seems to be too late for some."

Colour changes: Yellow Conservatives and blue Liberals

Oliver Letwin

David Cameron's policy chief, he warned campaign strategists early on to tone down the traditionalist Conservative rhetoric on immigration. A champion of the "Vote Blue, Go Green" environmental policy agenda.

Ed Llewellyn

Chief of staff to Cameron as opposition leader and now in Downing Street, Llewellyn worked for Lord Ashdown in Bosnia and is a friend of Nick Clegg's wife Miriam from their time working for the European Commission in Brussels.

Francis Maude

The minister in the Major government was a proponent of modernisation before Cameron became an MP in 2001. He was believed to be an enthusiastic supporter of the Lib-Con coalition. Now is Cabinet Office Minister.

Steve Hilton

Cameron's head of strategy who once voted for the Green Party, Hilton has pushed the case for the "liberal Conservative" agenda including prioritising the environment and the "Big Society" over right-wing messages on immigration and spending cuts. The Lib-Con coalition must be his idea of heaven.

Nick Clegg

Despite treading a careful path between Labour and Conservatives during his leadership, the new Deputy Prime Minister called for a more sceptical position on the EU. He promised to prioritise tax cuts over public spending.

David Laws

Former banker and multimillionaire was targeted for defection by George Osborne and joked that his friends presumed he was already a Tory. The new chief secretary to the Treasury will have no qualms about being the axe man.

Norman Lamb

North Norfolk's MP was also an early target for defection. He was branded a moderniser by the party's left when he devised a plan to part-privatise the Royal Mail. Many thought he would get a ministerial role, but none has appeared.

Jeremy Browne

The son of a diplomat is now a Foreign Office minister. Since being elected in 2005 in the Conservatives' south-west England heartland, he has been one of the most vociferous supporters of reform to the party, favouring tax cuts over "wasteful" public spending.