Emit INDENT/OUTDENT tokens from Flex - c

Question

I have found a great example of handling Python-style INDENT/OUTDENT in Flex here: http://matt.might.net/articles/standalone-lexers-with-lex/
The code is:
%{
#include <stdlib.h>
#include <stdio.h>
#include <assert.h>
#define MAX_DEPTH 72
int nesting = 0 ;
unsigned int indent_stack[MAX_DEPTH] ;
unsigned int level = 0 ;
unsigned int first = 1 ;
void process_indent(char* line) ;
%}
PUNCT [-+*/=<>:]*
ID [A-Za-z_][A-Za-z0-9_]*
INT [0-9]+
%option noyywrap
%%
^[ ]*\n {/* Ignore blank lines. */}
^[ ]*[^ \n]+ {int last = yyleng - 1;
process_indent(yytext) ;
while ((last >= 0) &&
(yytext[last] != ' ')) {
unput(yytext[last]);
last-- ;}}
"(" {printf(" "); ECHO; nesting++ ; }
")" {printf(" "); ECHO; nesting-- ; }
{ID}|{INT}|{PUNCT} {printf(" ") ; ECHO;}
[ \r] {}
\n {}
<<EOF>> { process_indent("") ; return 0 ; }
%%
unsigned int white_count(char* line) {
unsigned int count = 0 ;
while (*line == ' ')
count++, line++ ;
return count ;
}
void process_indent(char* line) {
if (nesting)
/* Ignore indents while nested. */
return ;
unsigned int indent = white_count(line) ;
if (indent == indent_stack[level]) {
if (!first) printf(" ;") ; // EMIT END_STATEMENT
first = 0 ;
return ;
}
if (indent > indent_stack[level]) {
printf(" {") ; // EMIT BLOCK_START
assert(level+1 < MAX_DEPTH) ;
indent_stack[++level] = indent ;
return ;
}
while (indent < indent_stack[level]) {
--level ;
printf(" }") ; // EMIT BLOCK_END
}
assert(level >= 0) ;
}
int main(int argc, char* argv[]) {
indent_stack[0] = 0 ;
yylex() ;
printf("\n") ;
}
Basically, it converts code like this:
def f(x):
y = x * x
e = (m * c
* c)
if x > 0:
return e
else:
return y
To this:
def f ( x ) : { y = x * x ; e = ( m * c * c ) ;
if x > 0 : { return e } else : { return y } }
Which is exactly what I need. With one difference: I want the lexer to return the appropriate tokens instead of printing out the curly braces.
The problem is with the OUTDENTs. This happens in a while block in the process_indent function (meaning: there may be more than one block to "close"). And that means I would have to return more than one tokens at the same time - which is NOT possible with Lex/Flex.
Any ideas how I could change that to make it work? Are there any working examples with what I need?

Answer 1

Any ideas how I could change that to make it work? Are there any
working examples with what I need?
You could base your code on the Python lexer example included with RE/flex, see ptokens.l. This lexer fully supports Python's syntax rules such as indentation blocks, and implicit and explicit line joining which isn't possible with process_indent(). Multiple "OUTDENT" (RE/flex "dedent") are generated when multiple blocks close.
The example requires RE/flex as a more advanced version of Flex. But RE/flex is very much like Flex and accepts Flex specifications, so moving on to RE/flex should be easy.