0

Unable to find out how the C-language lexical analyzer would tokenize the declaration of a static array. Say, int i[3]= {1,2,3};

The lexical analyzer would need to differentiate between just an integer identifier i, and a static array of size 3.

Am confused how the lexical analyzer would make tokens.

Tried Holub's book, among others, but couldn't find anything.

jiten
  • 115
  • 4
  • 2
    I'm afraid that you've reached the wrong site. This site is for questions about how to *teach* CS, and while that inevitably means that some CS gets discussed here, the questions themselves should largely revolve around advice for teachers. (Even self-learners are allowed, but even then, questions here should not strictly be about CS. (See the [meta post](https://cseducators.meta.stackexchange.com/questions/23/are-questions-about-self-learning-on-topic) about that tag.) – Ben I. Jul 25 '23 at 00:10
  • No, I've not. Just, want to impart more than could receive with MTech based on 433 AIR in GATE, that too having got admission in 3rd waiting list; with name at the fag end, of the big list; & left earlier institute to join "much more" prestigious NIT-Warangal, in hope to learn more. Such sites should be more liberal, else darkness here is not easy to go. – jiten Jul 27 '23 at 00:48

1 Answers1

1

Generally speaking a lexer (tokenizer) isn't concerned with semantics, only with classifying symbols, so i is just an identifier without meaning. It is the semantic analysis that connects the symbols with meaning. So, in the lex analysis no "distinguishing" needs to be done and generally won't be (can't be) if the tokenizing is single pass without backtracking. When i is analyzed, even the left bracket hasn't yet been considered.

It is possible, of course, to combine lex analysis (words) and semantic (meaning) analysis but it isn't normally done.

At the end of lexical analysis you have a list of all of the symbols in the program, those known (int, say) and those user defined, such as identifiers.

Buffy
  • 35,808
  • 10
  • 62
  • 115