On a walk through the wet fields of Buckinghamshire, the comedian and presenter talks about drinking lager aged three, his dad’s excuse for abandoning his family and why leaving Oxford University was like being thrown from a boat