Jamie Dornan says he knew 'Fifty Shades of Grey' movies would come with 'baggage'