Why lex invokes yyerror while parsing comma separated values? - c

I am preparing a yacc/lex test program. The lexer is intended to read integer numbers (long), float numbers (double) and date times in a specific format(YYYYMMDD HHMM).
lexer.l
%{
#include <time.h>
#include "grammar.h"
void read_float_number(void);
void read_integer_number(void);
void read_date_YYYYMMDD_HHMM(void);
void yyerror(const char* msg);
%}
%%
/* SKIP BLANKS AND TABS */
[\t ] { ; }
/* YYYYMMDD HHMM DATE */
[12][09][0-9][0-9][0-1][0-9][0-3][0-9][ ][0-2][0-9][0-5][0-9] { read_date_YYYYMMDD_HHMM(); return DATETIME; }
/* FLOAT NUMBER */
[0-9]+\.[0-9]+ { read_float_number(); return FLOAT_NUMBER; }
/* INTEGER NUMBER */
[0-9]+ { read_integer_number(); return INTEGER_NUMBER; }
%%
/* READ FLOAT NUMBER */
void read_float_number(void) {
sscanf(yytext, "%lf", &yylval.float_number);
}
/* READ INTEGER NUMBER */
void read_integer_number(void) {
sscanf(yytext, "%ld", &yylval.integer_number);
}
/* READ YYYYMMDD HHMM DATE */
void read_date_YYYYMMDD_HHMM(void) {
/* DATETIME STRUCT TM */
struct tm dt;
char buffer[80];
/* READ VALUES */
sscanf(yytext, "%4d%2d%2d %2d%2d", &dt.tm_year, &dt.tm_mon, &dt.tm_mday, &dt.tm_hour, &dt.tm_min);
/* NORMALIZE VALUES */
dt.tm_year = dt.tm_year - 1900; /* NORMALIZE YEAR */
dt.tm_mon = dt.tm_mon - 1; /* NORMALIZE MONTH */
dt.tm_isdst = -1; /* NO INFORMATION ABOUT DST */
mktime(&dt); /* NORMALIZE STRUCT TM */
/* PRINT DATETIME */
strftime(buffer, 80, "%c %z %Z\n", &dt);
printf("%s\n", buffer);
/* COPY STRUCT TM TO YACC RETURN VALUE */
memcpy(&dt, &yylval.datetime, sizeof(dt));
}
/* YYERROR */
void yyerror(const char* msg) {
fprintf(stderr, "yyerror %s\n", msg);
exit(1);
}
grammar.y
The grammar is intended to parse this kind of lines (DATETIME,FLOAT,FLOAT,INTEGER):
20191201 17000,1.102290,1.102470,0
%{
#include <time.h>
#include <stdio.h>
%}
%union {
struct tm datetime; /* DATE TIME VALUES */
double float_number; /* 8 BYTES DOUBLE VALUE */
long integer_number; /* 8 BYTES INTEGER VALUE */
}
%token <datetime> DATETIME
%token <float_number> FLOAT_NUMBER
%token <integer_number> INTEGER_NUMBER
%%
lastbid_lastask: DATETIME ',' FLOAT_NUMBER ',' FLOAT_NUMBER ',' INTEGER_NUMBER { printf("MATCH %lf %lf %ld\n", $3, $5, $7); }
;
%%
int main(int argc, char *argv[]) {
yyparse();
return 0;
}
The makefile to build everything is as follows:
CCFLAGS = -std=c89 -c
YFLAGS = -d # Forces generation of y.tab.h
OBJS = lexer.o grammar.o
TARGET = readfile
readfile: $(OBJS)
cc $(OBJS) -std=c89 -ll -o $(TARGET)
grammar.h grammar.o: grammar.y
yacc $(YFLAGS) -ogrammar.c grammar.y
cc $(CCFLAGS) grammar.c
lexer.o: lexer.l grammar.h
lex -olexer.c lexer.l
cc $(CCFLAGS) lexer.c
clean:
rm -f $(OBJS) grammar.[ch] lexer.c
I run readfile but after parsing the DATETIME lex seems to invoke yyerror:
% ./readfile
20191201 170003296,1.102290,1.102470,0
Mon Feb 17 22:20:00 2020 +0100 CET
yyerror syntax error
Same for numbers:
% ./readfile
45.45
yyerror syntax error
% ./readfile
45
yyerror syntax error
But not for arbitrary text:
% ./readfile
abc
abc
Why is lex invoking yyerror? What is missing in the lex parsing code?

As far as I can see, your lexer never returns a ',' token. By default, (f)lex scanners print unrecognised characters to stdout, as, for example, in your test with input abc. However, the unrecognised comma is not shown in your output because the stdout buffer was not flushed before exit() was called in yyerror().
In any event, we usually put a fallback rule as the last rule in the scanner specification:
. { return yytext[0]; }
That guarantees that any unrecognised character will be passed through to the parser as a quoted single-character token. If the parser does not expect that token, it will raise a syntax error immediately.

Related

How to resolve a yylloc undeclared error?

I'm new to flex and bison so bear with me. I'm trying to use yylloc in yyerror to print out where the error occurs along with the filename. I know that this requires me to redefine YYLTPYE to include a char* filename that I can use to keep track of the filename. According to the Flex and Bison book I have, it recommends that I use the YY_USER_ACTION macro to initialize the YYLTYPE in the .l file, so I included the following in it,
#define YY_USER_ACTION yylloc.filename = filename; yylloc.hel = 0; \
yylloc.first_line = yylloc.last_line = yylineno; \
yylloc.first_column = yycolumn; yylloc.last_column = yycolumn+yyleng-1; \
yycolumn += yyleng;
but when I try to compile the project, I get the error that yylloc is undeclared.
I've tried the solution offered by Chris Dodd in this question, but it hasn't helped me to resolve the issue. Any and all help in resolving this error is much apprecaited.
Here's the full code in .l:
%option noyywrap nodefault yylineno case-insensitive
%{
#include "need.h"
#include "numbers.tab.h"
int yycolumn = 1;
#define YY_USER_ACTION yylloc.filename = filename; yylloc.hel = 0; \
yylloc.first_line = yylloc.last_line = yylineno; \
yylloc.first_column = yycolumn; yylloc.last_column = yycolumn+yyleng-1; \
yycolumn += yyleng;
%}
Integers [-]?(0|[1-9][0-9]*)
Float [.][0-9]+
Exp [eE][-]?(0|[1-9][0-9]*)
Octal [-]?(00|0[1-7][0-7]*)
Hexa [-]?(0[xX][0-9A-F]+)
tomsNotNumbers [^ \t\n\v\f\r]+
%%
{Integers}{Float}?{Exp}? {
printf("%s is a number.\n", yytext);
possibleNumbers++; // increment by 1 as an input was given -M
actualNumbers++; // increment by 1 as an input did match our pattern -M
}
{Octal} {
printf("%s is a number.\n", yytext);
possibleNumbers++; // increment by 1 as an input was given -M
actualNumbers++; // increment by 1 as an input did match our pattern -M
}
{Hexa} {
printf("%s is a number.\n", yytext);
possibleNumbers++; // increment by 1 as an input was given -M
actualNumbers++; // increment by 1 as an input did match our pattern -M
}
{tomsNotNumbers} {
printf("%s is not a number.\n", yytext);
yyerror(warning, "This isn't a number.");
possibleNumbers++; // increment by 1 as an input was given -M
failedNumbers++; // increment by 1 as the input has failed to match our patterns -M
}
[\n] /*Do nothing for newline*/
. /*Do nothing for anything else*/
%%
.y is just empty for now, only has an include for need.h and one for .tab.h
The need.h:
#include <stdlib.h>
#include <stdarg.h>
#include <string.h>
int possibleNumbers = 0;
int actualNumbers = 0;
int failedNumbers = 0;
typedef struct YYLTYPE
{
int first_line;
int first_column;
int last_line;
int last_column;
char *filename; /* use to keep track of which file we're currently in */
int hel; /* no errors = 0, warning = 1, error = 2, fatal = 3 */
} YYLTYPE;
char *name; /*using for test purposes*/
# define YYLTYPE_IS_DECLARED 1
# define YYLLOC_DEFAULT(Current, Rhs, N) \
do \
if (N) \
{ \
(Current).first_line = YYRHSLOC (Rhs, 1).first_line; \
(Current).first_column = YYRHSLOC (Rhs, 1).first_column; \
(Current).last_line = YYRHSLOC (Rhs, N).last_line; \
(Current).last_column = YYRHSLOC (Rhs, N).last_column; \
(Current).filename = YYRHSLOC (Rhs, 1).filename; \
(Current).hel = YYRHSLOC (Rhs, 1).hel; \
} \
else \
{ /* empty RHS */ \
(Current).first_line = (Current).last_line = YYRHSLOC (Rhs, 0).last_line; \
(Current).first_column = (Current).last_column = YYRHSLOC (Rhs, 0).last_column; \
(Current).filename = NULL; \
(Current).hel = 0; \
} \
while (0)
typedef enum errorSeverity
{
warning = 1, error, fatal
} errorLevel;
void yyerror(errorLevel errlvl, char *s, ...)
{
va_list ap;
va_start(ap, s);
char *errLvls[3] = {"Warning", "Error", "Fatal"};
fprintf(stderr, "%s: %s: , %n", name, errLvls[errlvl - 1], yylloc.first_line);
vfprintf(stderr, s, ap);
fprintf(stderr, "\n");
}
main(int argc, char **argv)
{
printf("argv[0] = %s, argv[1] = %s.\n", argv[0], argv[1]);
if(argc > 1)
{
if((yyin = fopen(argv[1], "r")) == NULL)
{
perror(argv[1]);
exit(1);
}
name = argv[1];
} else
name = "(stdin)";
printf("Filename1: %s", name);
yylex();
printf("Filename2: %s", name);
// print out the report. -M
printf("Out of %d possible numbers, there were %d numbers, and %d not numbers.\n", possibleNumbers, actualNumbers, failedNumbers);
}
Since yylloc is normally defined in the bison-generated parser, not having a bison input file is going to be a bit of a nuisance.
Bison will define yylloc in the generated parser, and place a declaration in the generated header file, if:
You include the directive %locations in the bison prologue, or
You reference a location (#n for some n) in any bison action.
It is generally preferred to add the directive in case there is no explicit reference to a location in any rule.
As Chris Dodd says in the linked question, it is important to include the definition of YYLTYPE before #includeing the bison-generated header file. Alternatively, you could insert the definition of the structure, or an appropriate #include, directly in the bison prologue in a %code requires section. %code requires sections are copied to the generated header, so that will obviate the need to worry about the definition in the flex file.
By the way, I think you meant to use YY_USER_INIT to initialize yylloc. The expansion of YY_USER_INIT is executed only once, before the flex scanner's own initialization. The expansion of YY_USER_ACTION is executed before every scanner action (including empty actions), and is likely to be of use to update the yylloc structure with the current token.

Using parsers in GNU automake in c

I am new to GNU autotools and in my project lex and yacc parsers are used.Including them as source in makefile.am produes following error :
configure.in :
...
AC_CHECK_PROGS(YACC,bison yacc,none)
if test "x$YACC" = "xbison"; then
YACC="$YACC -y"
fi
AC_CHECK_PROGS(LEX,flex,none)
...
makefile.am :
## $Id
AUTOMAKE_OPTIONS=foreign no-dependencies
include $(srcdir)/Makefile_defs
dynamicpreprocessordir = ${libdir}/snort_dynamicpreprocessor
dynamicpreprocessor_LTLIBRARIES = libsf_appid_preproc.la
libsf_appid_preproc_la_LDFLAGS = -export-dynamic -module #XCCFLAGS#
if SO_WITH_STATIC_LIB
libsf_appid_preproc_la_LIBADD = ../libsf_dynamic_preproc.la
../libsf_dynamic_utils.la $(LUA_LIBS)
else
nodist_libsf_appid_preproc_la_SOURCES = \
../include/sf_dynamic_preproc_lib.c \
../include/sf_ip.c \
../include/sfPolicyUserData.c \
../include/sfxhash.c \
../include/sfghash.c \
../include/sflsq.c \
../include/sfhashfcn.c \
../include/sfmemcap.c \
../include/sfprimetable.c
libsf_appid_preproc_la_LIBADD = $(LUA_LIBS)
endif
libsf_appid_preproc_la_CFLAGS = -DDYNAMIC_PREPROC_CONTEXT -DSTATIC=static $(LUA_CFLAGS)
libsf_appid_preproc_la_SOURCES = $(APPID_SOURCES)
all-local: $(LTLIBRARIES)
$(MAKE) DESTDIR=`pwd`/../build install-dynamicpreprocessorLTLIBRARIES
In Makefile_defs :
APPID_SRC_DIR = ${top_srcdir}/src/dynamic-preprocessors/appid
...
APPID_SOURCES = \
$(APPID_SRC_DIR)/vfml/fc45.lex \
$(APPID_SRC_DIR)/vfml/fc45.y \
...
when i run the program i get following error :
libsf_appid_preproc.so: undefined symbol: FC45SetFile
While FC45SetFile() is already defined in fc45.lex file.
fc45.lex :
%{
#include "fc45.tab.h"
//#include "vfml.h"
#include <string.h>
#include <stdlib.h>
/* HERE doesn't match strings starting with numbers other than 0 right */
char string_buf[4000]; /* BUG - maybe check for strings that are too long? */
char *string_buf_ptr;
void FC45FinishString(void);
extern int gLineNumber;
%}
%x str_rule
%%
<str_rule,INITIAL>\|[^\n]* ;
[\ \t\r]+ ;
\n gLineNumber++;
\. { return '.';}
, { return ',';}
: { return ':';}
ignore { return tIgnore; }
continuous { return tContinuous; }
discrete { return tDiscrete; }
[^:?,\t\n\r\|\.\\\ ] string_buf_ptr = string_buf; unput(yytext[0]); BEGIN(str_rule);
<str_rule>[:,?] FC45FinishString(); unput(yytext[0]); return tString;
<str_rule>\.[\t\r\ ] FC45FinishString(); unput(yytext[1]); unput(yytext[0]); return tString;
<str_rule>\.\n FC45FinishString(); unput(yytext[1]); unput(yytext[0]); return tString; gLineNumber++;
<str_rule><<EOF>> {
int len = strlen(string_buf);
// printf("eof rule.\n");
if(len == 1 && string_buf[0] == '.') {
//printf(" period at end of file\n");
return '.';
} else if(string_buf[len - 1] == '.') {
// printf(" period: %s - unput .\n", string_buf);
FC45FinishString(); unput('.'); return tString;
} else {
// printf(" no-period: %s\n", string_buf);
FC45FinishString(); return tString;
}
}
<str_rule>\\: *string_buf_ptr++ = ':';
<str_rule>\\\? *string_buf_ptr++ = '?';
<str_rule>\\, *string_buf_ptr++ = ',';
<str_rule>\\. *string_buf_ptr++ = '.';
<str_rule>\n *string_buf_ptr++ = ' '; gLineNumber++;
<str_rule>[ \t\r]+ *string_buf_ptr++ = ' ';
<str_rule>[^:?,\t\n\r\|\.\\\ ]+ {
char *yptr = yytext;
while(*yptr) {
*string_buf_ptr++ = *yptr++;
}
}
%%
int fc45wrap(void) {
return 1;
}
void FC45SetFile(FILE *file) {
fc45in = file;
yyrestart(fc45in);
}
void FC45FinishString(void) {
int len;
char *tmpStr;
BEGIN(INITIAL);
*string_buf_ptr = '\0';
len = strlen(string_buf);
/* remove any ending spaces */
while(string_buf[len - 1] == ' ') {
string_buf[len - 1] = '\0';
len--;
}
tmpStr = MNewPtr(len + 1);
strncpy(tmpStr, string_buf, len + 1);
fc45lval.string = tmpStr;
string_buf[0] = '\0';
}
fc45.y :
%{
#include <stdio.h>
#include <stdlib.h>
#include <malloc.h>
#include "ExampleSpec1.h"
//#include "AttributeTracker.c"
//#include "vfml.h"
%}
%{
int fc45lex(void);
int fc45error(const char *);
/* HERE figure out how to give better error messages */
/* BUG needs a \n at the end of the names file */
/* These tmps are allocated at the begining of parsing and then
used during parsing. For example, so that we can simply
add terrains to an area while parsing. After parsing a
statement, the associated tmp is added to the appropriate
global list, and a new tmp is allocated. Finally, at the
end of parsing, all the tmps are freed
*/
ExampleSpecPtr exampleSpec;
AttributeSpecPtr attributeSpec;
int gLineNumber;
%}
%union {
int integer;
float f;
char *string;
}
%token <integer> tInteger
%token <string> tString
%token tIgnore tContinuous tDiscrete tEOF
%%
ExampleSpec: ClassList '.' AttributeList;
ClassList: ClassList ',' ClassSpec | ClassSpec /* ending */;
ClassSpec: tString { ExampleSpecAddClass(exampleSpec, $1); };
AttributeList: AttributeList AttributeSpec | /* ending */;
AttributeSpec: tString ':' AttributeInfo '.' {
AttributeSpecSetName(attributeSpec, $1);
ExampleSpecAddAttributeSpec(exampleSpec, attributeSpec);
attributeSpec = AttributeSpecNew();
};
AttributeInfo: tIgnore {
AttributeSpecSetType(attributeSpec, asIgnore);} |
tContinuous {
AttributeSpecSetType(attributeSpec, asContinuous);} |
tDiscrete tString {
AttributeSpecSetType(attributeSpec, asDiscreteNoName);
AttributeSpecSetNumValues(attributeSpec, atoi($2)); } |
AttributeValueNameList {
AttributeSpecSetType(attributeSpec, asDiscreteNamed); };
AttributeValueNameList: AttributeValueNameList ',' tString {
AttributeSpecSetNumValues(attributeSpec,
AttributeSpecGetNumValues(attributeSpec) + 1);
AttributeSpecAddValue(attributeSpec, $3); } |
tString {
AttributeSpecSetNumValues(attributeSpec,
AttributeSpecGetNumValues(attributeSpec) + 1);
AttributeSpecAddValue(attributeSpec, $1); };
%%
void FC45SetFile(FILE *file);
int fc45error(const char *msg) {
fprintf(stderr, "%s line %d\n", msg, gLineNumber);
return 0;
}
ExampleSpecPtr ParseFC45(const char *file) {
FILE *input;
input = fopen(file, "r");
if(input == 0) {
return 0;
}
FC45SetFile(input);
exampleSpec = ExampleSpecNew();
attributeSpec = AttributeSpecNew();
gLineNumber = 0;
if(fc45parse()) {
/* parse failed! */
fprintf(stderr, "Error in parsing: %s\n", file);
}
fclose(input);
/* free the left over attribute spec */
AttributeSpecFree(attributeSpec);
return exampleSpec;
}
I've searched Internet for the solution and was unable to come up with any.
Hope someone recognizes the problem and has a quick solution to it. Any help will be appreciated.
I just ran through a simple example, adding the following Autotools files:
configure.ac:
AC_PREREQ([2.69])
AC_INIT([example], [0.1a], [example#example.com])
AC_CONFIG_SRCDIR([ex1.l])
# Used only to shorten the otherwise lengthy compilation line in the output below.
AC_CONFIG_HEADERS([config.h])
AM_INIT_AUTOMAKE([foreign])
# I used C instead of C++.
AC_PROG_CC
AM_PROG_LEX
AC_PROG_YACC
AC_CONFIG_FILES([Makefile])
AC_OUTPUT
Makefile.am (taken pretty much straight from the Automake manual):
BUILT_SOURCES = ex1.h
AM_YFLAGS = -d
bin_PROGRAMS = ex1
ex1_SOURCES = ex1.l ex1.y
It turned out that you can't name the files with the same base file name. In my case, I had ex1.l and ex1.y. Using ex1_SOURCES = ex1.l ex1.y resulted in the following output when I invoked make:
make[1]: Entering directory '/home/kit/ex1'
/bin/bash ./ylwrap ex1.l lex.yy.c ex1.c -- flex
make[1]: Leaving directory '/home/kit/ex1'
make all-am
make[1]: Entering directory '/home/kit/ex1'
gcc -DHAVE_CONFIG_H=1 -I. -g -O2 -MT ex1.o -MD -MP -MF .deps/ex1.Tpo -c -o ex1.o ex1.c
ex1.l:5:17: fatal error: ex1.h: No such file or directory
#include "ex1.h"
^
compilation terminated.
Makefile:378: recipe for target 'ex1.o' failed
make[1]: *** [ex1.o] Error 1
make[1]: Leaving directory '/home/kit/ex1'
Makefile:281: recipe for target 'all' failed
make: *** [all] Error 2
Note the fact that flex was invoked in the second line, but bison/yacc was not. What's the reason? Well, the ylwrap script is the reason:
ex1.c: ex1.l
ex1.c: ex1.y
Because the script renames the output of ex1.l from "lex.yy.c" to "ex1.c", the makefile thinks that ex1.c is already built, so it won't do anything with the bison/yacc file, which means that ex1.h isn't built either.
You can't disable the ylwrap script, but you can work around it: simply rename your flex source file and change the references to the file name in your Makefile.in and configure.ac files as necessary. You shouldn't need to rename your bison/yacc source file because that would mean changing every #include "fc45.h" to #include "fc45_g.h" (or whatever you renamed the file to) in every C file as well as your flex source file.
The problem is that neither make nor automake know anything about the non-standard .lex file extension, so when you have a source file that ends in .lex, they don't know what to do with it. You could create rules to handle .lex, but it's probably much easier to just rename the file with a .l file extension, which they know how to handle.

Sqlite3 adding custom function

I'm trying to add a custom rank function written in C for SQLite3. The code compiles, but I can't seem to load it in sqlite3. When I do .load './librank.so' in SQLite3, I see this error message:
./librank.so.so: cannot open shared object file: No such file or directory
The code (from http://www.sqlite.org/fts3.html#appendix_a)
Rank.c
#if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_RANK)
#include <stdio.h>
#ifndef SQLITE_CORE
#include "sqlite3ext.h"
SQLITE_EXTENSION_INIT1
#else
#include "sqlite3.h"
#endif
static void rankfunc(sqlite3_context *pCtx, int nVal, sqlite3_value **apVal){
int *aMatchinfo; /* Return value of matchinfo() */
int nCol; /* Number of columns in the table */
int nPhrase; /* Number of phrases in the query */
int iPhrase; /* Current phrase */
double score = 0.0; /* Value to return */
assert( sizeof(int)==4 );
/* Check that the number of arguments passed to this function is correct.
** If not, jump to wrong_number_args. Set aMatchinfo to point to the array
** of unsigned integer values returned by FTS function matchinfo. Set
** nPhrase to contain the number of reportable phrases in the users full-text
** query, and nCol to the number of columns in the table.
*/
if( nVal<1 ) goto wrong_number_args;
aMatchinfo = (unsigned int *)sqlite3_value_blob(apVal[0]);
nPhrase = aMatchinfo[0];
nCol = aMatchinfo[1];
if( nVal!=(1+nCol) ) goto wrong_number_args;
/* Iterate through each phrase in the users query. */
for(iPhrase=0; iPhrase<nPhrase; iPhrase++){
int iCol; /* Current column */
/* Now iterate through each column in the users query. For each column,
** increment the relevancy score by:
**
** (<hit count> / <global hit count>) * <column weight>
**
** aPhraseinfo[] points to the start of the data for phrase iPhrase. So
** the hit count and global hit counts for each column are found in
** aPhraseinfo[iCol*3] and aPhraseinfo[iCol*3+1], respectively.
*/
int *aPhraseinfo = &aMatchinfo[2 + iPhrase*nCol*3];
for(iCol=0; iCol<nCol; iCol++){
int nHitCount = aPhraseinfo[3*iCol];
int nGlobalHitCount = aPhraseinfo[3*iCol+1];
double weight = sqlite3_value_double(apVal[iCol+1]);
if( nHitCount>0 ){
score += ((double)nHitCount / (double)nGlobalHitCount) * weight;
}
}
}
sqlite3_result_double(pCtx, score);
return;
/* Jump here if the wrong number of arguments are passed to this function */
wrong_number_args:
sqlite3_result_error(pCtx, "wrong number of arguments to function rank()", -1);
}
int sqlite3rankInit(sqlite3 *db){
sqlite3_create_function(db, "rank", -1, SQLITE_UTF8, 0, &rankfunc, 0, 0);
return 0;
}
#if !SQLITE_CORE
int sqlite3_extension_init(
sqlite3 *db,
char **pzErrMsg,
const sqlite3_api_routines *pApi
){
SQLITE_EXTENSION_INIT2(pApi)
return sqlite3rankInit(db);
}
#endif
#endif
Compiled with: gcc -shared -fPIC -lm rank.c -o librank.so
Can someone help me see what I did incorrectly?
Try to include the assert header
diff --git a/rank_orig.c b/rank.c
index 5d51b4d..4940e1b 100644
--- a/rank_orig.c
+++ b/rank.c
## -1,5 +1,6 ##
#if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_RANK)
#include <stdio.h>
+#include <assert.h>
#ifndef SQLITE_CORE
#include "sqlite3ext.h"
Compile:
$ gcc -shared -fPIC -lm rank.c -o librank.so
Load
$ sqlite3
SQLite version 3.8.10.1 2015-05-09 12:14:55
sqlite> .load ./librank.so
sqlite>
The error message shows that SQLite already appends ".so" to the file name.
So you must specify the bare library name:
> .load './librank'

incompatible types to type 'Ev' from type 'int' at line 50

Scheduler.c
I am making scheduling program for class and I am stuck with this error. It says incompatible types to type 'Ev' from type 'int'. I am confused to what this could mean.
int main()
{
Ev *event;
int numEvents=0;
int option;
int x;
int y;
char menu[]="1 - Insert a new event\n2 - Display all events\n3 - Now?\n4 - Delete expired\n0 - Exit\nPlease select an option: ";
event = malloc(sizeof(Ev) *numEvents);
CopynumEvents(&numEvents);
for (y=0; y<numEvents; y++)
{
event = realloc(event, sizeof(Ev) * (y + 1));
**MY ERROR SEEMS TO BE AROUND THIS AREA**
event[y]=CopyfromFile(numEvents,y);
}
option = MenuOption(menu);
while(1)
{
switch(option)
{
case(1):
event = realloc(event, sizeof(Ev) * (numEvents + 1));
numEvents++;
event[numEvents - 1]=InsertEvent();
ArrangeEvents(event,numEvents);
CopytoFile(numEvents, event);
option = MenuOption(menu);
break;
case(2):
if (numEvents!=0)
{
for (x=0; x<numEvents; x++)
{
DisplayEvent(event[x]);
}
}
else
{
printf("No event in scheduler\n");
}
option = MenuOption(menu);
break;
case(3):
ActiveEvents(event, numEvents);
option = MenuOption(menu);
break;
case(4):
DeleteExpired(event, &numEvents);
CopytoFile(numEvents, event);
option = MenuOption(menu);
break;
case(0):
free(event);
exit(1);
break;
}
}
}
int MenuOption(char *prompt)
{
char buffer[100];
int option;
do
{
printf("%s", prompt);
/* Get a line of up to 100 characters */
fgets(buffer, sizeof(buffer), stdin);
/* Remove any \n we may have input */
if(strlen(buffer) > 0)
buffer[strlen(buffer)-1] = '\0';
sscanf(buffer, "%d", &option);
}while(option!=1 && option!=2 && option!=3 && option!=4 && option!=0);
return option;
}
struct Event InsertEvent()
{
struct Event E1;
printf("What is the event: ");
InputEventname(E1.name, sizeof(E1.name));
E1.date = InputDate("Event date: ");
E1.start = InputTime("Start time: ",E1.date);
do
{
E1.end = InputTime("End time: ",E1.date);
}while(E1.end<=E1.start);
return E1;
}
void InputEventname(char *str, int max)
{
char buffer[100];
/* Get a line of up to 100 characters */
fgets(buffer, sizeof(buffer), stdin);
/* Remove any \n we may have input */
if(strlen(buffer) > 0)
buffer[strlen(buffer)-1] = '\0';
/* Copy up to max characters to our string */
strncpy(str, buffer, max);
str[max-1] = '\0';
}
time_t InputTime(char *prompt, time_t date)
{
char buffer[100];
char *result;
struct tm time;
time = *localtime(&date);
do
{
printf("%s", prompt);
/* Get a line of up to 100 characters */
fgets(buffer, sizeof(buffer), stdin);
/* Remove any \n we may have input */
if(strlen(buffer) > 0)
buffer[strlen(buffer)-1] = '\0';
result = strptime(buffer, "%I:%M%p", &time);
} while(result == NULL);
return mktime(&time);
}
time_t InputDate(char *prompt)
{
char buffer[100];
char *result;
struct tm date;
do
{
printf("%s", prompt);
/* Get a line of up to 100 characters */
fgets(buffer, sizeof(buffer), stdin);
/* Remove any \n we may have input */
if(strlen(buffer) > 0)
buffer[strlen(buffer)-1] = '\0';
result = strptime(buffer, "%m/%d/%Y", &date);
} while(result == NULL);
/* Convert to time_t format */
date.tm_min = 0;
date.tm_hour = 0;
date.tm_sec = 0;
date.tm_isdst = 1;
return mktime(&date);
}
here is my header. I believe I have done this part right
#include <stdio.h>
#include <stdlib.h>
#include <math.h>
#include <stdbool.h>
#include <string.h>
#define __USE_XOPEN
#include <time.h>
#include "scheduler.h"
/*
* This is my scheduling program!!
*/
/* Our possible scheduling states */
#define NewEvent 1
#define AllEvents 2
#define Now 3
#define Expired 4
#define Exit 0
#ifndef DISPLAY_H
#define DISPLAY_H
typedef struct Event
{
char name[50];
time_t date;
time_t start;
time_t end;
int y;
}Ev;
int MenuOption(char *prompt);
struct Event InsertEvent();
void InputEventname(char *str, int max);
time_t InputDate(char *prompt);
time_t InputTime(char *prompt, time_t date);
void DisplayEvent(Ev event);
#endif
The macro __USE_XOPEN is an internal name; it shouldn't be set by you. You should use #define _XOPEN_SOURCE 700 or something similar (500, 600, or 700). Further, it should be set before any POSIX header is included. The standard C headers are also POSIX headers.
You should give the exact error message (except for the pathname of the file(s) if they're longer than about 20 characters; chop them down to the basename) in the question. A key component of the information is the line number; you should identify that line in the file unless it is line 1 (maybe up to line 10, but don't omit any lines if you don't mark the line with the error).
Since you say the compiler warns about:
incompatible types to type 'Ev' from type 'int'
you could have a problem with malloc() and realloc() not being declared, or it could be CopyfromFile() that is not declared. If you don't declare a function before it is used, the C compiler will assume it returns an int — that was correct for C89 and pre-standard C. Strictly, for C99 or C11, the compiler should reject the compilation, but most don't unless you tell them to do so.
If you're compiling with GCC, use stringent warning options — especially to warn you about missing prototypes. I use:
gcc -O3 -g -std=c11 -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes \
-Wold-style-definition -Werror …
You might use -Wold-style-declaration too, or instead, and some (older) versions of GCC may not support the -Wold-style-* options. If your code compiles cleanly with those options, it avoids an awful lot of problems.

String input to flex lexer

I want to create a read-eval-print loop using flex/bison parser. Trouble is, the flex generated lexer wants input of type FILE* and i would like it to be char*. Is there anyway to do this?
One suggestion has been to create a pipe, feed it the string and open the file descriptor and send to the lexer. This is fairly simple but it feels convoluted and not very platform independent. Is there a better way?
The following routines are available for setting up input buffers for scanning in-memory strings instead of files (as yy_create_buffer does):
YY_BUFFER_STATE yy_scan_string(const char *str): scans a NUL-terminated string`
YY_BUFFER_STATE yy_scan_bytes(const char *bytes, int len): scans len bytes (including possibly NULs) starting at location bytes
Note that both of these functions create, return a corresponding YY_BUFFER_STATE handle (which you must delete with yy_delete_buffer() when done with it) so yylex() scan a copy of the string or bytes. This behavior may be desirable since yylex() modifies the contents of the buffer it is scanning).
If you want avoid the copy (and yy_delete_buffer) using:
YY_BUFFER_STATE yy_scan_buffer(char *base, yy_size_t size)
sample main:
int main() {
yy_scan_buffer("a test string");
yylex();
}
See this section of Flex's manual for information on how to scan in-memory buffers, such as strings.
flex can parse char * using any one of three functions: yy_scan_string(),
yy_scan_buffer(), and yy_scan_bytes() (see the documentation). Here's an example of the first:
typedef struct yy_buffer_state * YY_BUFFER_STATE;
extern int yyparse();
extern YY_BUFFER_STATE yy_scan_string(char * str);
extern void yy_delete_buffer(YY_BUFFER_STATE buffer);
int main(){
char string[] = "String to be parsed.";
YY_BUFFER_STATE buffer = yy_scan_string(string);
yyparse();
yy_delete_buffer(buffer);
return 0;
}
The equivalent statements for yy_scan_buffer() (which requires a doubly null-terminated string):
char string[] = "String to be parsed.\0";
YY_BUFFER_STATE buffer = yy_scan_buffer(string, sizeof(string));
My answer reiterates some of the information provided by #dfa and #jlholland, but neither of their answers' code seemed to be working for me.
Here is what I needed to do :
extern yy_buffer_state;
typedef yy_buffer_state *YY_BUFFER_STATE;
extern int yyparse();
extern YY_BUFFER_STATE yy_scan_buffer(char *, size_t);
int main(int argc, char** argv) {
char tstr[] = "line i want to parse\n\0\0";
// note yy_scan_buffer is is looking for a double null string
yy_scan_buffer(tstr, sizeof(tstr));
yy_parse();
return 0;
}
you cannot extern the typedef, which make sense when you think about it.
The accepted answer is incorrect. It will cause memory leaks.
Internally, yy_scan_string calls yy_scan_bytes which, in turn, calls yy_scan_buffer.
yy_scan_bytes allocates memory for a COPY of the input buffer.
yy_scan_buffer works directly upon the supplied buffer.
With all three forms, you MUST call yy_delete_buffer to free the flex buffer-state information (YY_BUFFER_STATE).
However, with yy_scan_buffer, you avoid the internal allocation/copy/free of the internal buffer.
The prototype for yy_scan_buffer does NOT take a const char* and you MUST NOT expect the contents to remain unchanged.
If you allocated memory to hold your string, you are responsible for freeing it AFTER you call yy_delete_buffer.
Also, don't forget to have yywrap return 1 (non-zero) when you're parsing JUST this string.
Below is a COMPLETE example.
%%
<<EOF>> return 0;
. return 1;
%%
int yywrap()
{
return (1);
}
int main(int argc, const char* const argv[])
{
FILE* fileHandle = fopen(argv[1], "rb");
if (fileHandle == NULL) {
perror("fopen");
return (EXIT_FAILURE);
}
fseek(fileHandle, 0, SEEK_END);
long fileSize = ftell(fileHandle);
fseek(fileHandle, 0, SEEK_SET);
// When using yy_scan_bytes, do not add 2 here ...
char *string = malloc(fileSize + 2);
fread(string, fileSize, sizeof(char), fileHandle);
fclose(fileHandle);
// Add the two NUL terminators, required by flex.
// Omit this for yy_scan_bytes(), which allocates, copies and
// apends these for us.
string[fileSize] = '\0';
string[fileSize + 1] = '\0';
// Our input file may contain NULs ('\0') so we MUST use
// yy_scan_buffer() or yy_scan_bytes(). For a normal C (NUL-
// terminated) string, we are better off using yy_scan_string() and
// letting flex manage making a copy of it so the original may be a
// const char (i.e., literal) string.
YY_BUFFER_STATE buffer = yy_scan_buffer(string, fileSize + 2);
// This is a flex source file, for yacc/bison call yyparse()
// here instead ...
int token;
do {
token = yylex(); // MAY modify the contents of the 'string'.
} while (token != 0);
// After flex is done, tell it to release the memory it allocated.
yy_delete_buffer(buffer);
// And now we can release our (now dirty) buffer.
free(string);
return (EXIT_SUCCESS);
}
Other-way, you can redefine function YY_INPUT in lex file, and then set your string to LEX's input. As below:
#undef YY_INPUT
#define YY_INPUT(buf) (my_yyinput(buf))
char my_buf[20];
void set_lexbuf(char *org_str)
{ strcpy(my_buf, org_str); }
void my_yyinput (char *buf)
{ strcpy(buf, my_buf); }
In your main.c, before scanning, you need to set lex's buffer first:
set_lexbuf(your_string);
scanning...
here is a small example for using bison / flex as a parser inside your cpp code for parsing string and changing a string value according to it
(few parts of the code were removed so there might be irrelevant parts there.)
parser.y :
%{
#include "parser.h"
#include "lex.h"
#include <math.h>
#include <fstream>
#include <iostream>
#include <string>
#include <vector>
using namespace std;
int yyerror(yyscan_t scanner, string result, const char *s){
(void)scanner;
std::cout << "yyerror : " << *s << " - " << s << std::endl;
return 1;
}
%}
%code requires{
#define YY_TYPEDEF_YY_SCANNER_T
typedef void * yyscan_t;
#define YYERROR_VERBOSE 0
#define YYMAXDEPTH 65536*1024
#include <math.h>
#include <fstream>
#include <iostream>
#include <string>
#include <vector>
}
%output "parser.cpp"
%defines "parser.h"
%define api.pure full
%lex-param{ yyscan_t scanner }
%parse-param{ yyscan_t scanner } {std::string & result}
%union {
std::string * sval;
}
%token TOKEN_ID TOKEN_ERROR TOKEN_OB TOKEN_CB TOKEN_AND TOKEN_XOR TOKEN_OR TOKEN_NOT
%type <sval> TOKEN_ID expression unary_expression binary_expression
%left BINARY_PRIO
%left UNARY_PRIO
%%
top:
expression {result = *$1;}
;
expression:
TOKEN_ID {$$=$1; }
| TOKEN_OB expression TOKEN_CB {$$=$2;}
| binary_expression {$$=$1;}
| unary_expression {$$=$1;}
;
unary_expression:
TOKEN_NOT expression %prec UNARY_PRIO {result = " (NOT " + *$2 + " ) " ; $$ = &result;}
;
binary_expression:
expression expression %prec BINARY_PRIO {result = " ( " + *$1+ " AND " + *$2 + " ) "; $$ = &result;}
| expression TOKEN_AND expression %prec BINARY_PRIO {result = " ( " + *$1+ " AND " + *$3 + " ) "; $$ = &result;}
| expression TOKEN_OR expression %prec BINARY_PRIO {result = " ( " + *$1 + " OR " + *$3 + " ) "; $$ = &result;}
| expression TOKEN_XOR expression %prec BINARY_PRIO {result = " ( " + *$1 + " XOR " + *$3 + " ) "; $$ = &result;}
;
%%
lexer.l :
%{
#include <string>
#include "parser.h"
%}
%option outfile="lex.cpp" header-file="lex.h"
%option noyywrap never-interactive
%option reentrant
%option bison-bridge
%top{
/* This code goes at the "top" of the generated file. */
#include <stdint.h>
}
id ([a-zA-Z][a-zA-Z0-9]*)+
white [ \t\r]
newline [\n]
%%
{id} {
yylval->sval = new std::string(yytext);
return TOKEN_ID;
}
"(" {return TOKEN_OB;}
")" {return TOKEN_CB;}
"*" {return TOKEN_AND;}
"^" {return TOKEN_XOR;}
"+" {return TOKEN_OR;}
"!" {return TOKEN_NOT;}
{white}; // ignore white spaces
{newline};
. {
return TOKEN_ERROR;
}
%%
usage :
void parse(std::string& function) {
string result = "";
yyscan_t scanner;
yylex_init_extra(NULL, &scanner);
YY_BUFFER_STATE state = yy_scan_string(function.c_str() , scanner);
yyparse(scanner,result);
yy_delete_buffer(state, scanner);
yylex_destroy(scanner);
function = " " + result + " ";
}
makefile:
parser.h parser.cpp: parser.y
# /usr/local/bison/2.7.91/bin/bison -y -d parser.y
lex.h lex.cpp: lexer.l
# /usr/local/flex/2.5.39/bin/flex lexer.l
clean:
- \rm -f *.o parser.h parser.cpp lex.h lex.cpp
There's this funny code in libmatheval:
/* Redefine macro to redirect scanner input from string instead of
* standard input. */
#define YY_INPUT( buffer, result, max_size ) \
{ result = input_from_string (buffer, max_size); }

Resources