Is a smaller triangle inside of a larger triangle where the outside of the smaller triangle and the inside of the larger triangle is the interior a polygon?

If at least one vertex of the inner triangle lies on the side of the outer triangle then yes, it is a complex polygon. However, if none of the vertices of the inner triangle are on the sides of the outer triangle then there are two disjoint triangles and that does not represent a polygon.