Can a british prime minister be removed at any time by his party?

No. A leader of a party can be removed by his party at certain times of the year, in the Conservatives I think it is November and Labour I believe the process starts before the conference in September, depending on the rules of that party. A Prime Minister can face a 'motion of no-confidence' in the House of Commons, and although there is no legal compulsion to leave office, it would create a constitutional crisis if a PM stayed-on after losing such a vote.