Is it harmful to pet rabbits if they live outside to bring them inside for a while during the winter?

Yes, if a rabbit starts to smell like a human, other rabbits will avoid him (or her). Since rabbits share food and huddle together for warmth, being avoided could be fatal (especially in Winter).