Does increasing of mass of rusty iron violate the low of conservation of mass?

No, because the principle of conservation of matter states that during a chemical reaction, matter is neither created nor destroyed. The total mass of reactants MUST equal the total mass of products.