Segmentation fault while convering large number of xml files into text - c

Question

#include<stdio.h>
#include<stdlib.h>
#include<dirent.h>
#include<string.h>
int main()
{
FILE *fin,*fout;
char dest[80]="/home/vivs/InexCorpusText/";
char file[30];
DIR *dir;
char c,state='1';
int len;
struct dirent *ent;
if((dir=opendir("/home/vivs/InexCorpus"))!=NULL)
{
while((ent=readdir(dir))!=NULL)
{
if(strcmp(ent->d_name,".") &&
strcmp(ent->d_name,"..") &&
strcmp(ent->d_name,".directory"))
{
len=strlen(ent->d_name);
strcpy(file,ent->d_name);
file[len-3]=file[len-1]='t';
file[len-2]='x';
//strcat(source,ent->d_name);
strcat(dest,file);
printf("%s\t%s\n",ent->d_name,dest);
fin=fopen(ent->d_name,"r");
fout=fopen(dest,"w");
while((c=fgetc(fin))!=EOF)
{
if(c=='<')
{
fputc(' ',fout);
state='0';
}
else if(c=='>')
state='1';
else if(state=='1')
{
if(c!='\n')
fputc(c,fout);
if(c=='.')
{
c=fgetc(fin);
if(c==' '||c=='\n'||c=='<')
{
fputc('\n',fout);
ungetc(c,fin);
}
else fputc(c,fout);
}
}
}
}
close(fin);
close(fout);
strcpy(dest,"/home/vivs/InexCorpusText/");
}
closedir(dir);
}
else
{
printf("Error in opening directory\n");
}
return 0;
}
I was trying to convert xml files to text. This code simply remove tags and nothing else.
When i execute this code for around 300 files, it doesn't show any error but when number goes to 500 or more i receive a segmentation fault after processing around 300 files.

Answer 1

At least one reason 'right from the start':
Here is struct dirent declaration from man:
On Linux, the dirent structure is defined as follows:
struct dirent {
ino_t d_ino; /* inode number */
off_t d_off; /* offset to the next dirent */
unsigned short d_reclen; /* length of this record */
unsigned char d_type; /* type of file; not supported
by all file system types */
char d_name[256]; /* filename */
};
You are in trouble on any name longer than 30 (actually 29) chars. Memory overwrite occurs because file has only 30 bytes (reserve 1 for '\0' terminator):
char file[30];
...
strcpy(file,ent->d_name);

Answer 2

There are two structures within XML that it does not appear that you account for.
Attribute contents can contain unescaped > characters, which could throw off your count. See http://www.w3.org/TR/REC-xml/#NT-AttValue.
CDATA sections can contain both < and > characters as literal text, as long as they do not appear as part of the closing ]]> string. See http://www.w3.org/TR/REC-xml/#NT-CharData. This could seriously throw off your logic.
Why don't you look in your files to see if any contain the text CDATA?
You might want to consider using xsltproc or libxslt; a very simple XSLT transform would give you exactly what you want. See Extract part of an XML file as plain text using XSLT for such a transform engine.

Answer 3

OK, another problematic place:
len=strlen(ent->d_name);
....
file[len-3]=file[len-1]='t';
file[len-2]='x';
Because d_name could have less than 3 characters it could again lead to memory overwrite.
You should be careful with functions like strlen() and always validate their result.