Can sperm inserted with the finger get you pregnant even though you start your period the next morning?

Highly doubtful, although especially in women with irregular cycles, pregnancy can result from the introduction of sperm almost anytime. This is why the idea that a woman can't get pregnant during her period is a myth. Also, implantation of an embryo can produce some bleeding, although there is generally a delay of several days between fertilization and implantation. Don't rule out anything.