What did the know-nothings organized to form what?

They formed the Order of the Star Spangled Banner to organize resistance against the Immigration of Roman Catholics (and more specifically, the immigration of Irish Catholics) into the USA. Although the movement had its heyday in the middle of the 19th century, it had a long-lasting influence on voter sensibilities: in 1960 John F. Kennedy's major electoral drawback in his bid for the Presidency was the fact that he came from an Irish Catholic background.