If a guy after the first date asks to walk you home and you decline will he not ask for a second date?

I'd say I don't think it will make a difference to a nice guy who respects you. Surely declining a walk home is a safety issue that he will then know where you live and you have only been on one date. Any guy worth something would respect this surely? I've been on one date with a guy, he asked me to come to lunch at his home next time and I said no not this soon and I would hope he would respect this otherwise he is not really worth having. What's he after if he doesn't respect your safety?