Whose pencil is this or who's pencil is this?

The correct form is "Whose pencil is this?" "Whose" is a possessive pronoun used to ask about ownership. "Who's" is a contraction for "who is" or "who has," which would not make sense in this context.