The Dink Network

Help! I got Milder's hammer, escaped the tunnel, went back to his house but the guards chased me away. So I asked the old man I got the scroll from earlier. He told me to go north to the old house. I do and go through the tunnel, which takes me to the top of the island and I still can't get to Milder. The walkthroughs are no help. What do I do?