As the latest series of Doctor Who draws to a close, we talk to fourth Doctor Tom Baker about the show's return and more. "They could bring me back as the Master!" he tells Paul Simpson.

You seem to have a taste for the fantastical. Was that always your take on Doctor Who?

My preoccupation in Doctor Who was always with science fantasy. Science fiction people are altogether more serious, but science fantasy people are the more general ones, who are very friendly and affectionate. There’s no discipline in science fantasy. There’s no physics. It’s more like born-again Christianity, in that there’s no theology, just a good feeling.

I was brought up in a rabid Roman Catholic household, accepting miracles and extraordinary happenings, so my idea of science fantasy was always very akin to religion. We laugh at Scientologists who don’t believe in death, but no religion does. It’s a transient thing. The whole thing about religion is that it’s an eternity of life. All those wild things that I believed didn’t bother me at all.

Why do you think the show has lasted so long?

It was rooted entirely in the characters. The wonderful running joke was that we used to have our adventures, but I never quite got to where I was going. I never had those navigational co-ordinates – I kept getting them wrong. I used to quote the telephone number at Television Centre, even down to the extension number! They never complained about that at the BBC, but they must have known.

The BBC saw a lot of people around the time they chose me for the Doctor. They interviewed a lovely man called Graham Crowden, who’s a very dedicated actor. I think he might be interested, but they said they wanted a two-year commitment. People like Graham didn’t want to commit for two years, but I would have signed for 22!

Do you think you outstayed your welcome, then?

Maybe I stayed too long, but my reasoning was, “Why should I leave when I enjoy it so much?" I might have stayed even longer, but there were certain tensions between John [Nathan Turner, 80s Who producer] and me about how it should go. John naturally wanted to put his mark on it, so when I said I thought I should stop, he didn’t say no.

I thought it was very odd of John, who was so astute in many ways, to give it to Peter Davison. That's not to impugn his ability, but he already had a fictional identity with children [in All Creatures Great and Small], so they had to make two leaps there. But nobody failed as the Doctor.

Are you happy still to be recognised as the Doctor?

Living in France, I thought it would be nice to be anonymous, but after you’ve been Doctor Who – or played striker for England or whatever it is you do – it’s terrible to be deprived of it. When I'm back here, people shout “Hello Doctor!” off scaffolding, and bus drivers pull up and say, “Come on, Doctor, I’ll take you!” People stop me in the street and say very sweet and sometimes very touching things.

Someone said to me on the radio that I always seem to have the time of day for people, but I wouldn’t exist without them. Some of them even employ me! It's the ultimate accolade, now I'm 73, to be working for people like Matt Lucas and David Walliams, who are the children who watched me when I was doing Doctor Who.

What kind of touching things do people say to you?

Someone stopped me in Oxford Street once and said, “Tom Baker? When I was a boy, I was in care in North Wales. No-one cared for us and no-one liked us, but on Saturday nights you made life better for me.” I went to speak to him, but he was so moved at the thought of how terrible his life had been, and the sight of an old hero, that he couldn’t speak any more. As he went, there was that wonderful eloquence of a little touch. I love all that. When I looked back, he couldn’t turn round.

I remember a girl in Australia, too, whom I'd never met, writing to say she wanted to have a child by me and could I be there on February 2nd? She said, “It may not be fun, because I weigh 280lbs and I’m 56 year olds, but I’ve got to have your child.” I sent her a little card that said I was terribly busy. She wrote back and said she understood!

I only have good memories. Some are fun and some are grotesque, in the sense of people declaring their love for me, writing to me for years, or wanting to marry me, but they're all good memories.

But doesn't all that recognition hold you back as an actor?

I don’t consider myself to be an actor. People don’t want me to be. You don’t get people saying, “Here’s Brad Pitt in a production that’s so amazing you won’t believe it’s Brad Pitt.” If you did you'd have an empty house, because people want to see Brad Pitt. Susan Hampshire did The Lady in the Van, and all her fans from lovely places like Cheltenham and Malvern went to see her. In the play, the lady in the van never leaves the bloody stage. People were saying, “When is Susan Hampshire coming on?” and there she was, all the time. They were disappointed because they didn’t want to see her like that.

All that nonsense about being unrecognisable is ridiculous. I just filter my lines through another little facet of me. I don’t want to be unrecognised. I've only got a very limited middle range voice, and I’m old now and very recognisable to English audiences… So what’s left for me?

You could always do Celebrity Big Brother?

I’ve been asked to go on one of those reality shows, like the one where Germaine Greer walked off. I like her. I think she’s so funny, given a chance. I remember saying to my wife that if I’d been on there, I would have said to her, out of the corner of my mouth, “Let’s get a scene going here.” Perhaps I could have persuaded her to pretend we were besotted with each other, but my wife wasn’t sure that was a very good idea!

Would you be interested in returning to Doctor Who one day?

I’d be very honoured and intrigued to do the new series. Obviously it occurred to me, because I’m not slow about those things. I haven’t seen any of it, but I've had nice messages from David Tennant, who obviously watched me, and I’m very glad that Sarah Jane [Elisabeth Sladen] was in it and has got a new series. That’ll generate a lot of new work for actors.

I thought it would be witty if they got me back as the Master, which would have allowed David to do a lot of double takes and say, “Haven’t I seen you somewhere before?” It's the old idea of Moriarty and Holmes being two sides of the same coin.

There may be something they come up with. Could one Doctor go and visit another? It would be a good commercial move and a good story! I like the idea that David might accidentally meet me, or have a terrible dilemma and have to seek me out. He’d say sorry to bother me, but he needs my help. I would live in a moving cottage, full of souvenirs from everywhere I’d been – signed portraits of Napoleon and letters of thanks from Alexander the Great. Maybe someone should speak to Russell about it!

Doctor Who series three concludes this Saturday at 7.05pm on BBC1. Robot, Tom Baker's first adventure as the Doctor, is out now on DVD.