Why isn't the fit perfect if the continents were once part of Pangaea?

Because the continents have distorted a little over time - as plates push together the land masses change. Also the coast has eroded over time.