DM pushing quest steps that don't relate to story

So in my game the story is revolving around a “Collector” who is trying to acquire one of my PC’s companions. The original quest was to enter the collectors manor and acquire a book but it changed when the Collector showed up at my PC’s manor and attempted to take his friend by force. The group came up with a plan to send the collector to PC’s patron and now suddenly everyone has forgotten that plan and are insisting we are going to infiltrate his manor.