What is it called when someone says they will do something and you say I bet you won't to try to make them?

The general term is provocation.

This is a type of dare that subtly challenges the person's courage, resolve or determination. Although the term "bet" is used, it may or may not result in an actual bet, or wager, on the outcome.

Whereas an expression of belief ("I'm sure you could") might not result in an attempt, the suggestion of fear or inability will often prompt a try, no matter how ill-advised or perilous the action. This is not as open a challenge as calling someone "chicken" (afraid or cowardly).