What does it mean when James Joyce said -History Stephan said Is a nightmare from which I am trying to awake?

The way I see it, Stephen is in fact expressing his frustration at his fellow countrymen, very preoccupied at that time with mourning an idyllic past, without taking the necessary steps in order to invent a glorious present or future. What Irishmen did (in Stephen's opinion, at least) was to complain and lament the British domination, at the same time betraying their leaders (the betrayal and subsequent death of Parnell influenced Joyce tremendously).

To better understand this affirmation I suggest you read something about The Irish Revival.