If your mother and stepfather owned a house and your mom died do the stepchildren have any rights to the house on your mother's behalf?

No, unless the property was titled in such a way that it prevents the automatic succession to the surviving spouse, which is highly unlikely. In all states a primary homestead by-passes probate procedure and becomes the sole property of the surviving spouse.