Struct Lexer 

pub struct Lexer<'a> { /* private fields */ }

Implementations

impl<'a> Lexer<'a>

pub fn new(input: &str, interner: &'a mut Interner) -> Lexer<'a>

Creates a new lexer for the given input text.

The lexer will tokenize natural language text according to the lexicon database, performing morphological analysis and ambiguity resolution.

Arguments

• input - The natural language text to tokenize
• interner - String interner for efficient symbol handling

Example

use logicaffeine_language::lexer::Lexer;
use logicaffeine_base::Interner;

let mut interner = Interner::new();
let mut lexer = Lexer::new("Every cat sleeps.", &mut interner);
let tokens = lexer.tokenize();

assert_eq!(tokens.len(), 5); // Quantifier, Noun, Verb, Period, EOI

pub fn tokenize(&mut self) -> Vec<Token>

Tokenizes the input text and returns a vector of Tokens.

Each token includes its type, the interned lexeme, and the source span for error reporting. Words are classified according to the lexicon database with priority-based ambiguity resolution.

Returns

A vector of tokens representing the input. The final token is typically TokenType::Eof.

pub fn is_collective_verb(lemma: &str) -> bool

pub fn is_mixed_verb(lemma: &str) -> bool

pub fn is_distributive_verb(lemma: &str) -> bool

pub fn is_intensional_predicate(lemma: &str) -> bool

pub fn is_opaque_verb(lemma: &str) -> bool

pub fn is_ditransitive_verb(lemma: &str) -> bool

pub fn is_subject_control_verb(lemma: &str) -> bool

pub fn is_raising_verb(lemma: &str) -> bool

pub fn is_object_control_verb(lemma: &str) -> bool

pub fn is_weather_verb(lemma: &str) -> bool