Meaning of Robert Frost poem In Neglect?

In Neglect THEY leave us so to the way we took, As two in whom they were proved mistaken, That we sit sometimes in the wayside nook, With mischievous, vagrant, seraphic look, And try if we cannot feel forsaken.

It's one of the poems where author/protagonist justifies his iconoclast. The path author follows is not accepted by others. He does it with his soul-mate, with him he drives the courage of defying all the challenges and gets the pleasure of returning the repulsion by becoming another pole.