The British actor, known for his role as Jackson Teller in Sons of Anarchy, was cast as BDSM-loving billionaire Christian Grey in the movie adaptation of EL James' bestselling erotic novels.
But when a promise made to friend and filmmaker Guillermo del Toro led to him leaving the franchise in October 2012 and being replaced by The Fall's Jamie Dornan, he was "heartbroken".
"It was the most emotionally destructive and difficult thing that I've ever had to deal with professionally," he told V Man magazine, explaining that it wasn't the kinky stripping off or pressure from fans that put him off.
"I'd given Guillermo my word, over a year before, that I was going to do Crimson Peak," he said. "People were saying, 'Are you crazy? Guillermo still has got four months to recast, it's the fourth lead, you can go and do this [instead].' I said, 'I can't. He's my friend.'"
Fifty Shades of Grey film stills
Show all 24
Hunnam tried his hardest to rearrange the schedules so he could shoot both films but the timetable proved a nightmare, with Sons of Anarchy producers also needing him on set for season seven.
The worst moment came when Hunnam had to tell director Sam Taylor-Johnson, who recently pulled out of doing the sequels, that he could not continue with the movie.
"I called her and we both cried our eyes out on the phone for 20 minutes," he said. "I needed to tell her that this was not going to work. There was a lot of personal stuff going on in my life that left me on real emotional shaky ground and mentally week. I just didn't know what to do."
Fortunately, there's not long to wait until horror-fantasy Crimson Peak reaches UK cinemas on 16 October.
Watch Apple TV+ free for 7 days
New subscribers only. £8.99/mo. after free trial. Plan auto-renews until cancelled
Whether it can surpass the massive $558 million that Fifty Shades grossed is unlikely, but the trailer vows it is del Toro's "ultimate masterpiece", so our expectations are high. Mia Wasikowska, Jessica Chastain, Tom Hiddleston and also star.
Join our commenting forum
Join thought-provoking conversations, follow other Independent readers and see their replies
Join our commenting forum
Join thought-provoking conversations, follow other Independent readers and see their replies