David Baddiel: So you think we've kicked racism out of English football?

He's only a poor little yiddo, he stands all alone on the Shelf

He's only a poor little yiddo,

He stands all alone on the Shelf,

He goes to the bar

To buy a lager

And only buys one for himself!

An old marching tune of Mosley's blackshirts? A rather literal translation of the second verse of the Horst Wessel song? Well, no actually: it was sung in London a little more recently than that. At Stamford Bridge, Chelsea Football Club's home ground, to be exact, the last time Chelsea played Tottenham, in spring 2002. As well as that ditty, there was:

Who's that team they call the Chelsea?

Who's that team they all adore?

Barcelona, Real Madrid,

Tottenham are a bunch of Yids!

Plus of course just the simple chant of "Yiddo! Yiddo!", repeated, in a deliberately guttural, threatening echo, with added ever-so-slightly Nazi finger movements, every five or six minutes.

This happens every time Chelsea play Tottenham. Chants of "Yiddo!" are also meted out to any player who spent any time at Tottenham, Teddy Sheringham, for example, when he was at Manchester United. I don't know how many Jews do actually support Tottenham (a fair amount, certainly), nor do I know how much this behaviour is replicated at other grounds, but I would be surprised if local derbies with Arsenal or West Ham are entirely Yiddo-free.

This week, after the appalling abuse Emile Heskey and Ashley Cole suffered at the hands of the Slovakian fans, there's been not a little self-congratulation about how well we've done in Britain to, as the saying goes, kick racism out of football. As a Chelsea fan and a Jew, some of this self-congratulation has left me feeling a little hollow.

Anti-Semitism is the "other" racism in football, and, because it so lags behind in visibility to racial abuse of black players, it is hardly ever discussed. But it's an important issue, not least because, in my experience, there are more Jewish fans at most grounds than black ones – it is virtually impossible to be Jewish and male and not interested in football – and I know that they all dread the starting up of these chants.

Without doubt, it seems mainly to be a London problem, and it does centre around Tottenham. For some time, in fact, I told myself that it didn't matter, that for most of these fans, "Yiddo" simply meant Tottenham player, or Tottenham fan, and that the negativity was just about that and not actually about race.

Then I heard the same chants shouted at Eyal Berkovic, Manchester City's Israeli midfielder, who has never played for Tottenham, and at the entire Hapoel Tel Aviv team when we played them in the Uefa Cup last season, and I realised that I was in denial about it: "Yiddo" may mean Tottenham fan, but it also means Jew.

Either way, who cares about exactly what the word means? If similar language were being used in en-masse singing and chanting, about Afro-Caribbeans or Indians, it would be a cause for national outrage. Certainly the club itself would be fined, and there would be an inquiry.

But I'm not blaming the club, there's little that it can do. The mass nature of the anti-Semitism – the fact that it exists in songs and chants – is the key to its survival. Abuse towards black players, thankfully, tends now to be individuals, odd shouts, easily targeted and dealt with. Chelsea has a policy, in fact, of encouraging its fans to report racism to the stewards, who should throw the racist out. But it's no good reporting the whole back half of the Matthew Harding Stand to the stewards.

I used to sit in that stand, until three seasons ago; the racism was one of the reasons I moved round to sit in the posher East Stand, where at least you can only hear the anti-Semites, rather than having to hug them when Chelsea score.

Having said that, the last time we played Tottenham, the bloke who sits next to me – who works in the music business – still got up and shouted "Palestine!", which I actually thought was pretty funny.

At some level, the anti-Semitism is funny. I now almost have a soft spot for "He's Only a Poor Little Yiddo", because it so demonstrates the small-minded nature of English racism, the only country in the world where the main crime anti-Semites would accuse Jews of is not drinking the blood of Christian babies, nor infiltrating the higher echelons of international finance, but failing to stand a proper round.

So sometimes it makes me laugh. But then, occasionally, when we're playing Tottenham, the hardest section of the crowd, the ones at the back of the Matthew Harding Stand, will start emitting an elongated hiss, supposed to emulate the sound of the gas chambers. And I'm not really sure how funny that is.

The writer is a comedian, novelist and football fan