Can you drive with the manifold not connected to the exhaust system?

Yes. You can. But there would be no point in an exhaust system after this modification. It would actually be in your disfavor to do this. Sure, it's louder. But you would then risk burning the gaskets and also Reduce your horsepower. Only do this temporary. But you should think about buying new headers (manifold) if they are damaged.