+1 credit
Video Upload Options
Lexical analysis is the initial level of natural language processing in which a text or speech-derived character sequence is transformed into computationally identifiable lexical units and their associated lexical information [1]. It encompasses orthographic processing and segmentation into tokens, together with the recognition and normalization of word forms, including the identification of lemmas, multiword expressions, and, where required, morphological structure and grammatical features [1][2]. The process maps surface strings to representations that can be matched against lexicons or inferred by linguistic and statistical models; it is therefore distinct from syntactic analysis, which determines structural relations among units, and semantic or discourse analysis, which interprets meaning beyond the lexical level [1][3].
References
- Max Silberztein. Lexical Analysis; Wiley: Hoboken, NJ, United States, 2016; pp. 213-255. [CrossRef]
- Daniel Jurafsky and James H. Martin. 2026. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models, 3rd edition. Online manuscript released August 19, 2026. https://web.stanford.edu/~jurafsky/slp3.
- Manning, C.D.; Schütze, H. 1999. Foundations of Statistical Natural Language Processing; MIT Press: Cambridge, MA, USA. Available online: https://bcmi.sjtu.edu.cn/home/zhaohai/lessons/nlp2011/textbooks/Foundations-of-Statistical-Natural-Language-Processing.pdf
