Artistic retreat: Vaughan Grylls' charming 14th-century haven in Kent

If you want to get some work done, head to the hills. Or, at least, out of the city. This was the artist Vaughan Grylls' plan when he and his wife Polly moved to their beautiful old house in "the middle of nowhere" in 1996. This nowhere being, more particularly, near Canterbury, "on the Ash Level" in Kent.

"Being an artist, it's useless if you have lots of people around you – you never get anything done," explains Grylls. "I've got to go away from things to get my head down. No one comes up to you to chat about contemporary art here – no one cares. You're incognito; it's not like being in Hoxton or Islington."

The listed building they call home dates back to 1386, although the pretty red-brick exterior is from, he believes, about 1702. It is set in a big open space, with farmland and orchards, and Grylls is a fan of the area: "One of the reasons we like it is that it's so un-Home Counties-like, you could be in Yorkshire."

It was love at first sight for the 68-year-old artist. His wife was hunting for properties, and they arranged to meet at the house. She didn't have any keys, so they made do with peering through the windows. Grylls didn't care: "I said 'I don't need to get in, this is amazing' – it's a peaceful, wonderful place! That was it really." Luckily for them, it was just as gorgeous on the inside, all thick wooden beams, stripped floorboards and cavernous fireplaces. The property's age, of course, means there are limits on the changes they can make anyway: "We did hardly anything. We're not allowed to do anything – not that we wanted to."

They did, however, take an axe to the leylandii trees surrounding it; those chunky logs stacked in a fireplace were once growing outside. "We had to chop them down and didn't know what to do with the wood – so some of them are still there. They look rather good, don't they?" he chuckles, before adding that they burn a treat, too: "If you put one of those logs on the fire on Christmas Eve it'll still be burning on New Year's Day."

Getting rid of the trees made room for a treasured possession to take pride of place on the drive. That red vintage vehicle is, he explains, "a very old Bristol car, an eccentric British classic. No one's ever heard of them – it's a bit like me!"

Of course, he's being modest. After studying at Goldsmiths and the Slade School of Fine Art, Grylls found acclaim as an artist in the late 1960s with his pieces dubbed "pun sculptures". He still has several of these – including the packing cases you can see him with here. Entitled A Case for Wittgenstein, they were part of his show at the ICA in 1969, after his graduation work was spied by art critic (and ICA assistant director) Jasia Reichardt. "I used to go to see Wittgenstein's lectures; of course I didn't understand them, but his picture theory of language made sense to me: he said, 'Statements are only meaningful if they can be pictured in the real world.'" And so, Grylls gave statements solid forms – in this case, two white suitcases purchased from BHS on Oxford Street.

Another piece, sat squatly on the floor of the living-room, next to that rather fine grand piano, is The Drunken Clergyman. Made in 1967, its appearance is halfway between a dog-collared vicar and an Irish coffee – and thereby hangs a tale…

When Grylls was eating at a Berni Inn, he encountered for the first time an exotic thing called Irish coffee. He drank several, and – feeling the effects – went next door into a churchyard and promptly fell asleep on a bench. "When I woke up hours later, I had a collar looking down on me," he says. "It was a vicar saying, 'Come on mate, up you get.'" This was the inspiration for the sculpture, which he aptly filled with liquid, so that when pushed, it makes a sloshing, slurping noise. It's the only sculpture he kept from that time, and it has moved with him ever since: "The last thing I did before I left my last home was put him in the front seat, and drive off."

Grylls achieved most recognition for his "joiner" technique: large-scale photo-collages joining separate photographs into a new whole. In 1978, his photo mosaic, Hagia Sophia, Istanbul, was chosen for an exhibition at the Whitechapel Gallery by its then-director, a young Nicholas Serota.

And Grylls continues to work in photo montage to this day. Behind him in his attic studio you can see a new work, being prepared for a show at the Piper Gallery, a space dedicated to artists whose careers span more than 40 years, which opened at the end of June. "The gallery has a serious point," says Grylls. "The idea of older artists being rich of experience, and that art tastes better when it is well-marinated – it's a simple but actually quite profound idea."

This latest piece is a large cross of black-and-white photographs. "It's made of photographs taken in St Petersburg, in this church where the last tsar's remains lie. Standing there was a woman who was explaining, in graphic detail, what had happened to the tsar to a deaf and dumb Spanish tour group – she was explaining in sign language." It is called – wait for it – Signs of the Cross. He may be a well-marinated artist, but Grylls' taste for puns shows no sign of weakening…

The 'Then and Now' exhibition is at The Piper Gallery, London W1 (thepipergallery.com), from today to 11 August