Why doesn't Paul want to leave before the Derby?

Paul doesn't want to leave before the derby because he is afraid that once he goes to seaside he will never get to come home to his little house again