What Difference between Token and lexeme?

1 answer

Answer

1161139

2026-03-02 16:30

+ Follow

One or many lexemes can belong to same token(category) and when lexeme recognized by a scanner to be in a some category that category returned as the token.

A lexeme is a section of text, which represents a token. For example in case of a number there are many lexemes representing the same token; for example: "12", "14.8" or "1001". Such general tokens are described by patterns of text.

ReportLike(0ShareFavorite

Copyright © 2026 eLLeNow.com All Rights Reserved.