pub struct Lexer<'a> { /* private fields */ }Implementations§
Source§impl<'a> Lexer<'a>
impl<'a> Lexer<'a>
Sourcepub fn new(input: &str, interner: &'a mut Interner) -> Self
pub fn new(input: &str, interner: &'a mut Interner) -> Self
Creates a new lexer for the given input text.
The lexer will tokenize natural language text according to the lexicon database, performing morphological analysis and ambiguity resolution.
§Arguments
input- The natural language text to tokenizeinterner- String interner for efficient symbol handling
§Example
use logicaffeine_language::lexer::Lexer;
use logicaffeine_base::Interner;
let mut interner = Interner::new();
let mut lexer = Lexer::new("Every cat sleeps.", &mut interner);
let tokens = lexer.tokenize();
assert_eq!(tokens.len(), 5); // Quantifier, Noun, Verb, Period, EOISourcepub fn new_imperative(input: &str, interner: &'a mut Interner) -> Self
pub fn new_imperative(input: &str, interner: &'a mut Interner) -> Self
A lexer for a fragment that is CODE, not prose — an LGX template hole, a directive
value, a filter argument. Ordinary Lexer::new starts in declarative mode and only
switches on a ## Main / ## To / ## Component header, which a bare fragment has
none of; a hole like todo's id or a . b must tokenize the imperative way.
Sourcepub fn tokenize(&mut self) -> Vec<Token>
pub fn tokenize(&mut self) -> Vec<Token>
Tokenizes the input text and returns a vector of Tokens.
Each token includes its type, the interned lexeme, and the source span for error reporting. Words are classified according to the lexicon database with priority-based ambiguity resolution.
§Returns
A vector of tokens representing the input. The final token is
typically TokenType::Eof.