What do all vertebrate have in common?

They all have similar nervous systems, including a spinal cord.

Obviously, they all have backbones/spines.

They all are chordates, (phylum chordata), which means that they have spines or at least spinal cords.

For more information about vertebrates, go to

http://www.ucmp.berkeley.edu/vertebrates/vertintro.html

For more information about chordates, go to

http://www.ucmp.berkeley.edu/chordata/chordata.html