Derren Brown looks deep into my eyes and smiles. ‘Being a hypnotist is the ultimate fantasy of control really, isn’t it?’

I can only agree with this master of manipulation, a grand illusionist with the power to read the minds of strangers and make them do outrageous things like rob a security van, shoot Stephen Fry or push an innocent victim off a roof. Those were faked for television specials The Heist, The Assassin and The Push (To The Edge) but the men and women involved though it was all happening for real.

So did the viewers who were alarmed when he risked blowing his brains out on television with Russian Roulette – or astonished and envious when he seemed to predict the right numbers on How To Win The Lottery.

So is that why Derren Brown became what he is: for the sense of power over people? ‘Big time. The desire to perform was huge, and so was the controlling aspect of it.’

The greatest mentalist magician of our age is making a new special for Netflix soon and touring Britain with a show called Underground celebrating his most successful stage tricks and illusions.

Then he’ll go to Broadway and attempt to seduce America.

Critics and audiences are always sworn to secrecy about his shows and Brown will never normally explain – except today is going to be different.

For once he will reveal some of his secrets and talk about a time it all went spectacularly wrong, live on stage. ‘I had nothing! I just had to say, ‘Sorry! Goodnight!’’

He’ll describe how one show got way out of hand when people started experiencing real ‘miracles’ even though he knew he was faking it. ‘There was one woman who had been paralysed all her life, she was in floods of tears, it was the first time she’d been able to move the side of her body.’

The success went to his head so much he thought about setting up as a real-life stadium healer. ‘I totally saw how you can start to go mad.’

And the great manipulator will warn that we are all in danger of being manipulated all the time by the politicians we elect to lead us.

’The more bewildered we are, the more suggestible we become.’

But what’s most fascinating – as we drink tea near his home in East London – is to see the mask slip and get a glimpse of the private emotions that motivate every trick and stunt that Derren Brown does.

Gone are the rakish frock coat, startled hair and devilish goatee with which he became famous at the start of the century. Today his head is shaven and he looks relaxed in jeans and a sloppy pink jumper.

Brown is out of his comfort zone, as he can’t control the questions, and keeps rubbing his head with apparent nerves. His eyes drill right through you though, when they are not restlessly scanning the room.

When I ask where he got this desire for power over people, Brown remembers his days as a shy, gay Christian pupil at a public school in Croydon, intimidated by the other boys as he struggled to reconcile his emerging sexuality with his faith.

‘I was not sporty in a fairly sporty school and not closeted but still working all that out. A step behind everyone else. They can sniff it out, can’t they? School is such an unforgiving environment.’

For self-defence, he acted larger than life and showed off with magic tricks. ‘I responded by being fairly intolerable, a terrible attention seeker. Then when I got to university, I saw this hypnotist perform, pretty early on. The people who tend to come up on stage at university gigs like that are the sporty types who really intimidated me back at school.’

So suddenly he saw the chance to get his own back on those jocks, by learning hypnosis and making them do something ridiculous in public like eat an onion thinking it’s an apple, or worse? ‘Exactly. Suddenly being seen as a powerful figure as opposed to a ridiculous figure was very appealing.’

He did need help though. ‘I did my first show at the end of my first year [studying law and German at Bristol University]. I mean, it was dreadful. Three hours long. I had trouble getting people up on stage so my Mum came up and played along. I remember that fondly.’

She was a model and Brown inherited her cheek bones. His father was a swimming teacher. Wasn’t their son scared of making a fool of himself, up on stage using techniques he had learned from books and only ever practised on friends? ‘I’m quite shy but I don’t have any nervousness about stepping on stage. As long as it’s something I have prepared. As long as I am on top of it.’

And Derren Brown has been on top of things ever since, creating a more elegant, even sinister persona for his debut Channel 4 show Derren Brown: Mind Control in 2000, in which he read the lives of complete strangers – from their habits, mannerisms, verbal ticks and other hidden ‘tells’ – before bending them to his will.

These days he deals in spectacle. Brown will sift through hundreds or thousands of volunteers to find the man or woman most open to his powers of suggestion, then place them unwittingly in an extraordinary situation: usually an elaborate set-up involving actors, a stunt crew and special effects. So a group of mild-mannered pensioners were somehow persuaded to steal a valuable painting in The Great Art Heist. A hesitant man called Matt was given the controls of a 737 passenger jet he thought was crashing and became – as the title suggests – a Hero At 30,000.

‘That made a real difference to him in real life,’ says Brown, who spent months and a small fortune creating a convincing zombie invasion for a different target, a self-confessed layabout called Stephen, who also discovered courage he didn’t know he had. ‘My favourite. I don’t know if the show can take all the credit, but Stephen’s life has changed a lot.’

So has Derren Brown, over the years. He’s as calm as you might hope from a man who has a best-selling ‘anti-self-help self-help book’ called Happiness: Why More Or Less Everything Is Absolutely Fine. Inspired by the ancient tradition of Stoicism, having shaken off his Christian faith long ago, Brown says: ‘It is not events out there that cause our problems, but our reactions to them.’

A cynic might say that’s easy to suggest when you’re a highly-acclaimed 47-year-old with an estimated £5 million in the bank, but Brown’s success was created by his insecurity. As a young man he hid shyness behind an act that only ended with fame.

‘I stopped being such a dick in real life when I started performing, because I would just funnel everything into that. Then once I started doing television, I stopped the need to ever perform in real life. I just grew out of it. I stopped doing tricks with people in real life. I’m a bit introverted and that’s fine.’

Still, everyone expects him to be able to see into their soul or take secret control of them and that’s exhausting. ‘I’ve got a good friend now who was convinced when we first met that I was controlling him – everything, every gesture, every turn of a cup.’

Brown leans forward as he says this, turns the paper tea cup that is sitting in front him with a very deliberate action and looks up with a sly smile, holding my gaze. ‘It took a few meetings before I convinced him. It’s so far from my mind, all of that, when I meet people.’

Is he controlling me now? He insists not. ‘When I go out and do a show, that’s the best, very charismatic version of me. That’s a lovely feeling. But doing it in real life all the time? That would be sad. Maybe I’m just reacting now against having done that for years.’

Is this reticence for real? ‘Yeah. Not all the time. I can easily go back to feeling a bit intimidated if I’m in a room with big characters, which can happen as you get well known. I do shrivel up a bit.’

One of those intimidating big characters used to be the host and comedian Stephen Fry, who he first met as an undergraduate. The memory makes him wince. ‘I met him at a book signing and he could not have been any nicer, but it was disappointing to me.’

He felt he knew Fry already, from having seen him on television, and at the back of his mind thought they would bond instantly, but of course to Fry he was just another face in the queue. ‘I think I expected we’d go out for dinner or something. I brought so much weight to that meeting, expectations which could never be met.’

Now Fry is a friend who even pretended to be shot on stage for a show back in 2011, and they do actually go out for dinner, but Brown still remembers what it felt like to be on the other side of fame in that first shy encounter. ‘I always try and be very present when I meet people.’

Rubbing his head again, he laughs apologetically. ‘I can’t remember what you asked me now … sorry!’

I press on, asking if his public persona makes it difficult to have a genuine relationship with a lover. ‘Not really. The guy I’m with now, Justin, was sort of aware of me when we met but not like a big fan or anything. That’s kind of nice. Dating a fan would be a bit odd.’

He is wary. ‘I have a few stalker fans: nothing terrible, but they like to find my partners and become best friends with them, or at least act like they are.’ Insinuating their way into his life by sucking up to his partner? That sounds scary, which explains why he is so protective of Justin and was of his previous long-term boyfriend, a graphic designer called Mark.

Brown was brought up with an Evangelical Christian faith that told him homosexuality was a sin and he even took part in a counselling course to try and ‘cure’ himself. He stopped believing in that (or any other) vision of God in his twenties, but was in his thirties before he came out as gay to family and friends. ‘I’m embarrassed a bit that it took so long,’ said Brown after revealing the truth publicly for the first time in 2007. ‘But if I hadn’t had a weird and slightly self-absorbed twenties then perhaps I wouldn’t be doing what I’m doing now.’

Has he ever used his powers to get a date? ‘For evil? Ha! No. I honestly don’t even know what it would mean to hypnotise someone into a date, or anything else. I just keep it on the stage.’

How does he feel about those self-styled pick-up artists who use psychology, people-reading and the power of suggestion to approach their ’targets’ on the street or in bars?

‘There’s that whole weird world of seduction isn’t there?’

Their techniques – revealed in a controversial book called The Game – are not so far away from some of what Brown does, surely? ‘No, they’re not and it’s so easy to sniff at them but I know somebody who absolutely used them and now has a wife and a couple of kids.’

Still, pick-up artists have been widely condemned as sexist and creepy and Brown is not a fan. ‘I think so many of those techniques and what passes off as hypnosis in that world is just doing stuff and not feeling guilty. Being unpleasant and cynical and not feeling bad about it.’

Some use an approach called neurolinguistic programming, which he went to a conference about. ‘There was a seat at the front which I went for and another guy went for it as well. He looked at me and said: ‘You will lose.’ So I politely said for him to sit down and he said to the guy next to him: ‘Did you see what I did?’ I just thought: ‘Is that hypnosis? Or is that just being a dick?’

Scrupulously polite, quiet and thoughtful off stage, Brown likes to read, paint and take street photographs. He lives in an ultra-fashionable part of East London with a couple of dogs, a live parrot called Rasputin and a menagerie of stuffed creatures, one of which he helpfully offers to our photographer as a prop: ‘Shall I get my penguin?’

Later this year he will try to break America, having tested the waters at a small theatre in New York last year. ‘I really loved it. Here you get a group reaction, everybody laughing or gasping together. There it was like two hundred individuals, each shouting on their own. ‘No way!’ Everybody’s turned up to 11.’

His Netflix special will also be based in America. ‘It’s very exciting. I can’t say what it is but I was tweeting a while back for American right-wingers and left-wingers, so politics certainly plays a part.’

One fan responded to this appeal by posting: ‘Please tell me you are re-running the Russian Roulette stunt but with six loaded chambers every time …’ That says a lot about our feelings towards politics these days and Brown is alarmed that we are all having our strings pulled by party advertisers and leaders including Donald Trump.

‘It’s a classic hypnotic technique to induce confusion then give a direct suggestion which someone is more likely to follow because it’s a relief. So politicians will give you a whole load of statistics and things you can’t quite follow, then say: ‘Therefore, we must do this …’ And you believe it a lot more because you’ve just heard all this stuff you haven’t quite followed.’

The claims made during the Brexit debate come to mind. ‘Yes. If we end up uncertain of everything, that can make us much more susceptible.’ Have we been conned by the Brexiteers? ‘We’ll find out, won’t we? It’s early days really, but we’ll see where it takes us.’

You might imagine his fame would make it harder to get British audiences to go along with him during live performances, but Brown says the opposite is true. ‘People are more suggestible if they know – or even just imagine – that I’m constantly doing stuff to influence them.’

This led to some astonishing results in his last show Miracle, as he sought to expose and recreate the way fake faith healers work. Like them he called people to the stage and offered instantaneous healing. ‘I imagined somebody might have a bit of a bad back and come up and say, ‘I feel better.’ The adrenaline would clear the pain, essentially. But it was so much more than that.’

Genuine healings appeared to happen, to his astonishment, including the paralysed woman. ‘Every night, someone with tinnitus would put their hand up and say it had gone. There was a guy with trigger finger, where your finger or your thumb lock. This 60-something guy just couldn’t believe it, he could move his fingers.’

What was really happening there? ‘I did nothing other than to create this environment whereby the guy with the trigger finger would stop telling himself the story of ‘I can’t move my fingers’. That’s all I’m doing. And it doesn’t mean that everyone with trigger finger is going to be healed. Maybe there’s something in people breaking the constantly negative cycle and doing something positive for a bit. Then the shock at feeling, ‘Oh God it’s not hurting!’ That gets the adrenaline going even more.’

How long would this last? ‘I imagine that for most people the effects wouldn’t last more than the ten minutes they were on stage, but I was getting letters and emails from a small percentage of people months later saying, ‘Look, this is still the same.’ Somebody said their husband had watched the show on TV and this golfing injury he’d had for 15 years cleared. He was too embarrassed to say so at the time, he just didn’t believe it, but a year later he said: ‘Just so you know, it’s been gone since and it was from watching the show.’ Amazing, isn’t it?’

‘I totally saw how you can start to go mad, to the point I was semi-seriously saying to the guys I work with: ‘I could do this at the O2 couldn’t I? I could be very upfront and say it’s nothing more than suggestion and adrenaline, but a certain percentage of people seem to have really been healed…’ You just start to go down that slippery slope into thinking you have some kind of special ability.’

So why didn’t he hire the O2? ‘No matter how carefully and honestly you advertised it, you’d have people turning up desperate for a real healing.’

He couldn’t behave like one of the charlatans he sees preying on people who still believe sincerely in a faith he used to hold dear.

But shouldn’t he be talking to the NHS about what he has learned? ‘That would be an interesting thing to talk about with professionals. Really, it comes back to bedside manner. The sad thing is, if you go and see a GP and get your allotted six minutes and he tells you to relax more, you’re going to feel unlistened-to. If you go to a holistic therapist you are there for an hour, the advice is essentially the same but you’ve been physically touched, you’ve gone through some sort of ritual and feel listened to. It’s all very important stuff … but I didn’t want to get into that world of having people turn up expecting that their problems will be solved.’

When he first started out, Derren Brown used to quote Sherlock Holmes a lot and he even lived in a £3.5m apartment on Baker Street. He shares the fictional sleuth’s uncanny ability to read people and pick up hidden signals, so shouldn’t he be helping the cops?

‘I did get an email from the police asking whether I’d come and talk to them. I’ve always avoided anything like that. I’m quite open about how the whole thing I do happens in inverted commas, so not to believe everything you see or hear. It’s a form of entertainment. Some of it’s real and some of it isn’t. Hopefully part of the fun is trying to unpick that.’

Nor has he been tempted by the corporate dollar. ‘There’s a whole world of business people who might want to learn these techniques to improve sales and so on, but it’s just not me. There are people who do that, but they are pretending. They’re passing off tricks as real stuff. That’s not my world and I don’t need to do it.’

Since he refuses to say which of his greatest hits are in the new show Underground, how about his biggest flop?

‘Oh God. There’s always things that don’t work live. The most excruciating was in the show Infamous. There was a running gag about not being able to mind read someone’s phone number like a bad psychic, just getting the zero and the seven then leaving it there.

‘Then at the very end of the show I’m getting numbers called out by the audience, I’m doing all this lightning calculation and the grand total of multiplication is the big reveal. ‘This is your phone number!’ Only, this one night, it wasn’t. Not even close!’

Brown had done this hundreds of times before and always managed to successfully predict and reveal the phone number of the person he had targeted. But this time, as a thousand people watched, he got it wrong.

’That was the very last moment of the show, the climax of this recurring theme we’d had for two hours, the cue for all the confetti to go off. And I was just really wrong. The audience was confused.’

Brown was totally exposed in front of all those people.

‘I had nothing. I just had to go, ‘Oh! Sorry! Well, goodnight!’’

They probably thought it was yet another mind game. ‘Yeah, they might have gone, ‘That certainly made him seem human.’ That’s always a plus point. But not during the finale!’

His laughter is edgy, because the old shyness is never far from the surface and the mistake caused him distress and soul-searching. But the answer was to go back over the show with forensic precision, examine what went wrong and make sure it never happened again. In other words, to take back control.

When he’s not on stage or in front of a camera, this great pretender certainly does seem human. Warm, self-aware, charming even. He’s given a little more of himself away than usual today.

But later, when I’m describing how Derren Brown was surprisingly good company, I think of the precise way he turned the cup as if it was some kind of mind control signal and the mysterious smile he gave me at that moment – and I wonder: ‘Is that what he told me to say?’

This is an extended version of the interview that first appeared in Event, the magazine of the Mail on Sunday

‘Derren Brown: Underground’ tours the UK from April 3 to July 5, derrenbrown.co.uk