Was Anne Boleyn Mary Queen of Scotts Sister?

No she was not. Anne Boleyn had a sister named Mary, but she was not the Scottish Queen.

Mary Queen of Scots was the great-niece of Henry VIII though so while Anne was married to him Anne would have been Mary's great-Aunt.