Nothing gives me as much pleasure as following in the footsteps of heroes whose exploits I have tucked under my arm. It's about crossing distance and twisting time, about physically connecting with something or someone through the potent mix of place and printed word.

I've retraced the prison exploits (real and imagined) of Henri "Papillon" Charrière in French Guiana. I've pursued the British imperial forces through Zululand, armed only with a Washing of the Spears by Donald Morris. And I've sat in the Antarctic hut where Apsley Cherry Garrard took shelter before his Worst Journey in the World.

Fiction works for me, too. In 2000 I travelled to Haiti with one of my favourite novels, The Comedians by Graham Greene.

In Port au Prince I stayed at The Hotel Oloffson, a weird gothic fantasy that Greene used as "The Trianon" in his thriller. The capital city is a small one if you're a blanc and it wasn't long before someone told me that a character in the book, the mercurial gossip columnist "Petit Pierre", was based on a journalist called Aubelin Jolicoeur – who happened to be drinking in the hotel bar.

I had to pinch myself as I drank gin and tonics with the tiny 80-year-old who'd been an apologist for (and survivor of) the murderous Duvalier regimes. "To a new friend," reads the shaky handwriting in the flyleaf of my Penguin, "because he has brought out the best of me". Actually I didn't; he was a notorious old flatterer as well as an egoist to whom I'd shown very real interest. But I still look at the inscription and marvel.

When you're caught up in the exploits of a hero – or anti-hero for that matter – a well-written text will provide a travel itinerary more compelling than any dreamed up by a tour operator. You you're steered by the pages, frequently going to the where via the when and the what.

That fourth-dimension moment occurs when it all meshes. It's sublime – if not always safe …

In 1998, my travels with the 1969 memoir Papillon began at the place where Parisian pimp Henri Charrière began his life sentence – beneath the arched gate of the Camp De La Transportation in St-Laurent-du-Maroni. The tin-roofed prison was a barely-tended ruin, rusting in a South American climate so fetid that it caused my book to swell with moisture. Only one in 10 lived to tell of the French colony's brutal camp, including inmate of concrete cell No. 47, Charrière.

However, while navigating via his page-turning tales of escape – through the jungles, along the soupy brown rivers and in outpost towns of French-speaking "bush negroes" – I learned how Charrière had taken the daring exploits of real-life escapees and recast them as his own. He'd collected them while doing orderly duties in the prison hospital, a privilege given to exemplary inmates.

My end-point of the trip was to be the pinnacle, a visit to the infamous solitary confinement on Devil's Island where Charrière was said to have escaped on a raft of coconuts. (This was another invention: he was never on Devil's Island, but did a spell on the neighbouring Ile Royale.)

A ferry went from the town of Korou to a resort on Ile Royale, which overlooked Devil's Island. I missed it by minutes. So I frantically petitioned yachties in the harbour, eventually persuading a lovely guy from Dutch St Maarten to sail me to the island for $50.

The four-hour trip was priceless, the sailor a true sea-spirit, and we came within metres of landing on Devil's Island – only to be intercepted by a French police helicopter which bent our yacht over with its downdraught and broadcast instructions that we were under arrest for stealing a yacht and should sail back to the mainland immediately.

After passports were confiscated I was questioned for five hours, guilty until proven innocent under French law. I gabbled about my wanting to follow in Papillon's footsteps and how I had pressed the sailor into helping us. My interrogator was baffled, then amused, then satisfied.

But the sailor – guilty only of working for a drunken boat owner who had decided to create a fuss – was kept locked up.

Later, I wrote:

On my release I asked to see Juliot [the sailor]. The gendarme led me to the cells, produced a large heavy key and slid two vast bolts on a heavy iron door. The light went on and the prisoner sat upright, eyes screwed shut against the light. The cell was stark, with concrete floor, wooden bench and no pillow. The open squat toilet had a ferocious stink. "Juliot… I'm really sorry," I began. I was shocked. The man of the sea looked frightened. And old. And worst of all – less of a man. "It'll be OK," I mumbled, "they've got the story straight." He looked at me, blinking behind his pebble glasses. I mumbled some more, I gave him my address. And to my surprise he gave me a hug. And then the door was shut and the bolts redrawn.

Bang. Bang.

The experience changed how I looked at Henri Charrière, a writer who was derided for cashing in on adventures that were not his own. I came out with even more respect for the man who endured 11 years of penal servitude and brought it to life with authority. If Charrière had used artistic license he'd damned well earned it.

Not all my heroes were so colourful.

I spent six months as a prospector in the  north-eastern goldfields of Western Australia and kept a suitcase of books in my tent, most of them written in the 1890s. One small book had me completely hooked – the diary of young New Zealander John Aspinall, posthumously published under the title And Some found Graves by AJ Thompson.

Aspinall went through the usual torments suffered by men of the goldfields, finding more hardship and comradeship than gold. He was no literary genius, but he was wide-eyed and engaging; he kept me company and helped me learn my craft, occasionally serving as an erstwhile treasure map (the old timers had uncanny abilities to break new ground).

But my companion's last entry was a poignant one. Friday 13, March 1896: "For the last few nights we have had some heavy thunderstorms …"

Storms in the desert pack an incredible punch and not long after this entry, Aspinall's body was found 100m from his camp, face-down, half his clothes torn away. His body was buried where it was found, on the Hawk's Nest goldfield near Laverton.

Talking over a campfire with a prospector mate, I discovered that he too had a fondness for Aspinall, a book much-loved by modern prospectors.

"You ever been to his grave?" I asked.

"No," he replied, "but I'd love to take a look."

We drove 200km from our bush camp, locating one of the goldfield's many "lonely graves" down a red dirt track. Under a hot sun among sparse mulga trees we stood by the tin headstone with its hand-written misspelling – "killed by lightening" – and drank a beer to John Aspinall. We left another beer beside the grave. We weren't the first.

My recent stay in the Chobe Game Lodge in Botswana was given a whole new dimension thanks to the diary entries of Richard Burton – and no, not the great Victorian explorer, but the melancholic Welsh actor who resolved to remarry Liz Taylor after an especially rough landing on the resort's airstrip in 1975. I was astonished not by how much had changed in 40 years but how little. The diary entries made it perfectly clear: he and I were sitting in the same boat looking at the same animals at the same bend on the same beautiful river. The only difference was I was a little more sober.

Today, there are heroes I'm still wanting to travel with. I want to visit Cuba in the company of The Old Man and the Sea – to the fishing village of Kojimar just outside Havana, where Hemingway's luckless old fisherman dreams of lions before doing three days of battle with his monster marlin. And I'd love to explore the village of Willingdon in England to see what, if anything, helped shape one of the most important books of the 20th century, George Orwell's Animal Farm.

How far will I travel with a hero? Well, there was one obsession that saw me clocking up thousands of miles – yet paradoxically I might have been better off if I'd stayed at home.

In the late 1990s, when I was working for a London newspaper, I became interested in the brilliant physicists working under J. Robert Oppenheimer in Los Alamos, New Mexico. It was all thanks to a story superbly told by Richard Rhodes in The Making of the Atomic Bomb.

I'd been to the genesis of the nuclear age – to Ernest Rutherford's den in Canterbury College in Christchurch, New Zealand, the homely timber spaces where the father-to-be of nuclear physics studied and conducted experiments. I'd been to the other end of the story – to the Pacific island of Tinian, to the bomb pits where the top secret weapon dubbed 'Little Boy" was loaded into the belly of a B52 ("Colonel," asked a subaltern who'd guessed at the mission, "are we splitting atoms today?"). And of course I went to the Peace Museum in Hiroshima, which stays in my memory for small children squealing with delight as they played chasey around exhibits of a city that was savagely vapourised.

In 2001 I finally made it to a windswept plain in New Mexico – the Trinity Bomb Site, open to visitors only on the first Saturday of April. Trinity is where "The Gadget" was hoisted on a 30m gantry in July, 1945, its small heart of uranium "warm like a rabbit".

I remember small kids from this trip, too – kids holding their parents' hands under a miserable cold wind, scuffing at wet sands complaining that there was "nuthin' t'see".

They were right: the gantry was just four nubs of concrete. Yet the pages of Rhodes' book filled the cold desert with extraordinary dramas of science, war and morality. I explored the ruins of an observation post where Oppenheimer, Enrico Fermi, James Chadwick and other laureates of physics watched The Gadget release its power – the moment when Oppenheimer said Vishnu's words came to him from the Bhagavad Gita: "I am become death, the destroyer of worlds".

For me, the ghosts of the scientists were as real as the cold wind blowing off the desolate Oscura Mountains.

Soon after that trip, I moved back to Australia to settle in a small village in the Adelaide Hills – and I learned that Sir Mark Oliphant had been resident in a village 4km up the road. Oliphant was one of Oppenheimer's inner circle of physicists, a founder of the Manhattan Project whose work on refining uranium made him a true father of the nuclear age – and he'd died just 12 months before I arrived.

To think I could have met him. If only I wasn't too busy travelling.