The Tokenizer type exposes the following members.
Constructors
Name | Description | |
---|---|---|
![]() |
Tokenizer |
Construct a tokenizer for a given input.
|
Methods
Name | Description | |
---|---|---|
![]() |
Equals | (Inherited from Object.) |
![]() |
Finalize |
Allows an Object to attempt to free resources and perform other cleanup operations before the Object is reclaimed by garbage collection.
(Inherited from Object.) |
![]() |
GetHashCode |
Serves as a hash function for a particular type.
(Inherited from Object.) |
![]() |
GetType |
Gets the Type of the current instance.
(Inherited from Object.) |
![]() |
MemberwiseClone |
Creates a shallow copy of the current Object.
(Inherited from Object.) |
![]() |
Scan |
Scans the input, breaking it down into tokens.
Returns null if attempts to read past end of the file
|
![]() ![]() |
ToString | Overloaded. |
Fields
Name | Description | |
---|---|---|
![]() |
currentLine | |
![]() |
currentLineNumber | |
![]() |
currentPos | |
![]() ![]() |
EOF |
A token which represents the end of file/input.
A TokenMatcher can return this symbol if it wishes to terminate processing of the input.
|
![]() ![]() |
EOL |
A token which represents and end of line.
|
![]() |
matchers | |
![]() |
sr | |
![]() |
tokens |