Who or what determines the date for inaugurations of presidents?

The US Constitution sets the date. Amendment 20 sets the beginning of the each for January 20. The body of the Constitution sets the term length at 4 years and gives Congress the power to set the date for the elections.