DocTokenizer
class DocTokenizer
Documentation comment tokenizer.
Functions
| Name | Description |
|---|---|
| DocTokenizer | Constructor. |
| next | Get the next token. |
| consume | Consume tokens until stopping criterion. |
Function Details
DocTokenizer
DocTokenizer(const TextLineCursor& source)
Constructor.
comment- Comment to tokenize.
consume
DocToken consume(const int stop = ANY)
Consume tokens until stopping criterion.
stop- Bitmask giving the token types on which to stop and return.
- Return
- Last token consumed.
Tokens are consumed until one is encountered with a type in stop, which is then returned. If stop is ANY then the next token is returned.
next
DocToken next()
Get the next token.
- Return
- Next token.
The token is only valid for the lifetime of the DocTokenizer, as it contains a reference to a substring of the source file. If no tokens remain (i.e. the end of the file is reached), a token with a type of NONE is returned.