If a player is outside the field of play in the vicinity of the penalty area beyond the byline and is fouled?

A foul cannot occur off the field of play. Any infractions off the field would be punishable as misconduct with a caution or a send off.

A referee could wait and deal with the misconduct at the next stoppage, in which case the reason for the stoppage would dictate the restart.

If a referee must deal with it immediately, a send off for example, then the restart will be an indirect free kick for the other team at the position of the ball at the moment of the stoppage.