Extracting text from big files in Prolog - file

Question

I would like to extract text between a start and end string with SWI-Prolog, e.g., all the titles from Wikipedia dumps. I don't want to use an XML-parser, as I want to deal with different file types in the same way. I got it working for small files, but run into problems for large files.
For big files (e.g., Romanian Wikipedia) prolog runs out of memory (prolog -G1G -L1G -T1G -s main.pl -t main, see content of main.pl below):
Welcome to SWI-Prolog (threaded, 64 bits, version 7.4.2)
SWI-Prolog comes with ABSOLUTELY NO WARRANTY. This is free software.
Please run ?- license. for legal details.
For online help and background, visit http://www.swi-prolog.org
For built-in help, use ?- help(Topic). or ?- apropos(Word).
found: 'Rocarta'
found: 'Muzică'
found: 'Iris (formație românească)'
found: 'Pagina principală'
...[removed hundreds of lines]
found: 'Zadar'
found: 'Australia'
found: 'Slovenia'
found: 'Croația'
ERROR: Out of global stack
Exception: (5,861) between([60, 116, 105, 116, 108, 101, 62], [60, 47, 116, 105, 116, 108, 101, 62], _264890370, [10, 32, 32, 32, 32, 60, 110, 115|...], []) ?
How to accomplish this task with big input files?
MWE (main.pl):
:- use_module(library(pio)).
:- use_module(library(dcg/basics)).
last_call_optimisation(true).
main :-
phrase_from_file(between(`<title>`, `</title>`, _), `wiki.xml`).
between(Start, End, Found) -->
string(_), string(Start), string(Found), string(End),
{ format("found: '~s' \n", [Found]) },
between(Start, End, _).
between(_, _, []) -->
remainder(_),
{ format("finished parsing") }.
example input (wiki.xml):
<mediawiki>
>< Don't use an XML parser! ><
<page><title>Albert Einstein</title></page>
<page><title>Elvis Presley</title></page>
</mediawiki>
example output (expected):
found: 'Albert Einstein'
found: 'Elvis Presley'
finished parsing
Edit:
If we remove the recursive call from between/3, the output changes, and doesn't correspond to what I expect:
found: 'Albert Einstein'
found: 'Albert Einstein</title></page>
<page><title>Elvis Presley'
found: 'Elvis Presley'
finished parsing

Answer 1

this construct
..., string(_), string(Start), ...
it's very inefficient. It turns a linear parse into an exponential one.
But we have a really simple solution, since a string literal performs an exact match in a DCG:
:- use_module(library(dcg/basics)).
main(Titles) :-
%phrase_from_file(between(`<title>`, `</title>`, Titles),`wiki.xml`).
phrase(between(`<title>`, `</title>`, Titles), `
<mediawiki>
>< Don't use an XML parser! ><
<page><title>Albert Einstein</title></page>
<page><title>Elvis Presley</title></page>
</mediawiki>
`).
between(_Start, _End, []) --> [].
between(Start, End, [Found|Rest]) -->
Start, string(String), End,
{ atom_codes(Found, String) },
!, between(Start, End, Rest).
between(Start, End, List) --> [_], between(Start, End, List).
I would simplify the code, though:
...
phrase(tag(`title`, Titles), `
...
tag(_Tag, []) --> [].
tag(Tag, [Found|Rest]) -->
"<", Tag, ">", string(String), "</", Tag, ">",
{ atom_codes(Found, String) },
!, tag(Tag, Rest).
tag(Tag, List) --> [_], tag(Tag, List).
My bet is that on large files this is slightly more efficient.
It's also easy to generalize:
...
phrase(tags([title, footnote], Contents), `
...
tags(_Tags, []) --> [].
tags(Tags, [Key-Found|Rest]) -->
"<", {member(Tag, Tags)}, Tag, ">", string(String), "</", Tag, ">",
{ maplist(atom_codes, [Found,Key], [String,Tag]) },
!, tags(Tags, Rest).
tags(Tags, List) --> [_], tags(Tags, List).
but not very efficient. Better (but should profile to prove it)
...
"<", string(Tag), ">", {memberchk(Tag, Tags)}, string(String), "</", Tag, ">",
...
Edit: at least on a small set of Tags, "<", {member(Tag, Tags)}, Tag, ">" seems to require a lot less inferences than "<", string(Tag), ">", {memberchk(Tag, Tags)},.