Does the outside temperature affect overall physical performance?

Yes it can. The body will use a lot of energy trying to stay warm or cool. This use of energy leaves less energy for the body to perform.