Is the Spanish word viajar masculine?

No. Nor is it feminine, either. It is the infinitive of a verb (= to travel), which doesn't have masculine and feminine in Spanish.