The black faces that adorn the canvas of history

What did Black Britain look like in the 1800s? There were perhaps 20,000 black people in Britain at the time of abolition in 1807. None were slaves. Slavery in the home country of the Empire had been gradually proscribed after the Mansfield judgment of 1772; because, as every pedant now knows, what was abolished 200 years ago was the British slave trade.

So black Britons were free men - they were overwhelmingly male - but their status was ambiguous. Most conspicuously, they served as valued members of the armed forces. The vast canvas of The Death of Nelson, by Daniel Maclise, casts a black seaman in a central role, pointing towards the sniper whose bullet had brought down the admiral.

Although it depicts the Battle of Trafalgar, 1805, it was painted half a century later, a commission for which Maclise carried out detailed research, even interviewing surviving members of the crew of Victory. The inclusion of two black faces in this tableau of Britishness was historically accurate. There were 17 nationalities aboard Victory, among them nine West Indians, one African and one Salvadorean.

A handful of black celebrities were also lauded in the early 19th century. George Bridgetower, for example, was a virtuoso violinist who was taken up by the Prince of Wales (the future George IV). He came to England as a child prodigy, having been born in Poland, the son of a black valet, possibly an escaped slave, who served an aristocratic family there. He toured Europe as a star turn on the courtly music circuit, becoming a friend of Beethoven and an original member of the Royal Philharmonic Society when it was founded in 1813.

Then there was Olaudah Equiano, who published The Interesting Narrative of the Life of Olaudah Equiano, the African, Written by Himself in 1789. It was one of several widely read accounts of life as a slave that helped to stir abolitionist sentiment, and he became a leading figure in the anti-slavery movement.

Others enjoyed more mundane success. George Africanus has recently been claimed as Britain's first black entrepreneur. As a boy he was given as a present to the Molineux family of Wolverhampton (after whom the Wanderers' ground is named). The family educated him, and he became an apprentice in one of its foundries. He moved to Nottingham, married a local woman, and set up an employment agency for domestic servants. It was profitable enough to make him a property owner and freeholder.

Linda Colley argues in Britons: Forging the Nation that a British identity was available to nearly any member of marginalised groups who proved their patriotism, and this was certainly true for many. But it was a fickle status. There was some solidarity between the black and white poor, living in the same parts of London that the most marginal groups have always lived - the East End and around the docks, as well as, in those days, St Giles. Sir John Fielding, a magistrate and brother of the novelist Henry Fielding, complained that when slaves ran away from their masters they often found "the mob on their side".

But there was hostility, too. Some of the examples of what would now be regarded as gross racism - the fashion for exotically dressed servants or freak-show entertainments - could be ascribed to the wonder of the uneducated at the novelty of relatively few black people. But there was also fear and dislike, too, of a kind that still echoes in attitudes to asylum-seekers today.

The defeat of the British in the American War of Independence in 1781 led to a sudden influx of black soldiers who had fought on our side. Although some black loyalists continued to serve in the armed forces, those who could not find work had a hard time. Denied Poor Relief, they were driven to begging.

A combination of the panic of the respectable classes and simple philanthropy prompted the founding of a Committee for the Relief of the Black Poor. The committee, seeing the black poor as a "burthen" on the (white) public, devised a scheme to ship as many as possible to Sierra Leone, a British colony in west Africa. Unsurprisingly, few could be persuaded to go. Eventually 309 black men, 41 black women and 59 white women, the wives or widows of black men, set sail in 1787. Most died of disease on arrival, and their settlement was destroyed in fighting between slave traders and a local ruler. After four years, only 60 had survived.

We have come a long way in 200 years. But as recently as the run-up to the last election the Conservatives devised a plan to process all asylum applications on an island "far, far away".