Is it better to have a smaller or equal size rim as the rear wheel while biking downhill and off-road on a mountain bike?

It's a personal preference thing. Some downhill specific mountain bikes have a 24" tire in the back as opposed to a 26" tire because there's less rotational mass so it's a little easier to accelerate. The 24" rim is also stiffer and more durable than a 26" wheel.

But as you hit obstacles on the way down the hill, the 24" wheel can feel more harsh. Also, tight turns can feel a bit weird with two different size wheels. Another problem with two different sized wheels is that you have to have two different sized spare tubes and tires because if you get a flat, you'll never know ahead of time if it's going to be the front or back.