The various token types are used to syntax highlight particular words in a language. This makes code easier to read. There is a wide latitude in the usage of the token types, and really it depends on the specifics of the language as to which token represents which type. Some examples are given below, but these are just guidelines, not hard and fast rules.
Many languages include constructs from other languages. One common example is html files can include javascript and css blocks. Several of the mode tags support a DELEGATE attribute, which will allow a section of text to be passed to a different mode for highlighting. The html mode delegates to the javascript mode for javascript blocks and to the css mode for style blocks. Use of the DELEGATE attribute is highly encouraged when appropriate since it makes writing modes easier, reduces duplication, and promotes visual consistency across languages.
Parser rules can highlight tokens using any of the following token types:
NULL - no special highlighting is
performed on tokens of type NULL
COMMENT1
COMMENT2
COMMENT3
COMMENT4
jEdit supports four different types of comment tokens. Generally, comments are programmer-readable constructs that are ignored by compilers and interpreters. As an example, the lisp mode defines four comment types:
<EOL_SPAN TYPE="COMMENT4">;;;;</EOL_SPAN>
<EOL_SPAN TYPE="COMMENT3">;;;</EOL_SPAN>
<EOL_SPAN TYPE="COMMENT2">;;</EOL_SPAN>
<EOL_SPAN TYPE="COMMENT1">;</EOL_SPAN>
FUNCTION
The function token is intended to identify functions, methods, procedures, routines, or named subprograms.
DIGIT
The digit token is to identify numbers.
INVALID
The invalid type is to indicate that particular words are not to be used, for example, the java mode defined both "goto" and "const" as invalid words. These are words that are defined by the language, but are not to be used.
KEYWORD1
KEYWORD2
KEYWORD3
KEYWORD4
Keywords are used to identify well-defined words within a language. Some languages naturally divide keywords into groups, for example, the pascal mode identifies "for" as a KEYWORD1, "private" as a KEYWORD2, and "int" as a KEYWORD3.
LABEL
A label is generally a named position within a source, for example, the ada mode defined a label as <<foo>>.
LITERAL1
LITERAL2
LITERAL3
LITERAL4
Literals are usually, but not always, uninterpreted strings, for example, "foo" or 'bar'. There are a wide variety of usages of literals in the mode files.
MARKUP
The markup token is generally used in the various "markup" languages, such as xml and html. Markup is used for those elements that are not specified as words belonging to the language. For example, in html, <body> would be considered a keyword, where <foo> would be considered markup.
OPERATOR
Common examples of operators are the math symbols, such as '+', '-', and so on.