Can we use the word ''courage'' instead of the word ''encourage'' in the sentence ''you are encouraging him''?

Not really. There may have been a time long ago when the Words meant the same thing (they do share a common ancestry), but these days, they carry two different meanings. The one, to encourage, means to boost someone's confidence, to cheer them on; it carries a meaning of directly saying or doing something that positively impacts the other person. I'm a Big Sister, and my Little Sister is very shy, so I try to encourage her by telling her she has a great personality.

Courage, on the other hand, carries a meaning of individual bravery, such as we might see in a soldier, a first responder (firefighter, police officer), or someone who is willingly taking a major risk. I have great admiration for Mahatma Gandhi: it took great courage to stand up to an angry mob and defend religious tolerance. Thus, in the sentence you are giving, the Word "courage" would not work, since you are trying to positive impact the other person-- which is the meaning of "encourage."