I really think either could work. For the most part, the only rougher terrain is between Marmora and Bancroft. Most of the other trails and gravel roads are well suited to a gravel bike. What kind of tires do you have on your gravel bike?
On second thought, you might overall find the route more enjoyable on a hard tail for some extra comfort. My route includes some gravel road options to break up the trail, but many riders just stay on the rail trail the whole time between Haliburton and Bancroft. Personally, I find that a bit tedious.