Do all states have a capitol building?

Yes, all states do have a capitol building. Some are called State Houses but serve the same purpose.