Function commands::tokenizer::tokenize
[−]
[src]
pub fn tokenize(text: &str) -> Result<Vec<Token>, TokenizerError>
Tokenize a body of text.
pub fn tokenize(text: &str) -> Result<Vec<Token>, TokenizerError>
Tokenize a body of text.