What will happen if I don't replace CV Boot?

If it's cracked, sand, etc. will get into the joint and it will wear out and fail. As the link providing power to the wheels, failure of the CV joint will leave the vehicle undriveable when it fails.

Depending on how much grit get into the joint, it may go a long time before failure, or it may not last long at all. Depending on your situation, you could tape up the hole or crack and wait until it goes out. Also, considering that replacing the boot can cost nearly as much as replacing the the entire halfshaft, replace the the entire halfshaft when you do it.