What prevented Europeans from exploring in the middle ages?

The lack of means to establish a ship's position in the open seas. There were primitve means for that and the Vikings had been good in using them, ennabling their ships to travel from Scandinavia to places like Iceland, Greenland and Newfoundland - and back to base. But the use of navigational instruments was not common elsewhere in Europe, causing most ships to stay close to the coast.

This did not so much 'prevent' exploring, but it did limit its scope. The Portuguese for instance had been exploring ever further south along the west African coast during the Middle Ages.