How long did it take to build Alcatraz?

Alcatraz had fortifications from the 1850s to the early 1900s, and was later turned into a Federal Pen. from 1934 to 1963

So, different structures were being built/demolished during an 160 year period.