Solr suggester dictionnary not building. Java heap space error? - solr

Question

I'm using solr 5.3.1 and sunspot 2.2.7 on a Rails API with PostgreSQL database.
I've been trying to configure an autosuggest/autocomplete feature for days but struggle to make it work. I want that looking for "foob" return the suggestion "foobar company".
My schema.xml contains this :
<copyField source="*_text" dest="textSpell" />
<copyField source="*_text" dest="autocomplete" />
<copyField source="*_s" dest="textSpell" />
this allow me to copy for the spellcheck (which works fine) and for the autocomplete from the dynamic solr field created by sunspot :
<dynamicField name="*_text" stored="false" type="text" multiValued="true" indexed="true"/>
This dynamicField contains the value I want to work with : title_text.
My fields for spellchecking and autocompletion looks like this :
<field name="textSpell" stored="false" type="textSpell" multiValued="true" indexed="true"/>
<field name="autocomplete" stored="true" type="autocomplete" multiValued="true" indexed="true"/>
My fieldType for autocomplete looks like this :
<fieldType name="autocomplete" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.EdgeNGramFilterFactory" minGramSize="1" maxGramSize="25" />
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
Then on solrconfig.xml i have my suggester components :
<searchComponent name="suggest" class="solr.SuggestComponent">
<lst name="suggester">
<str name="name">suggest</str>
<str name="lookupImpl">FuzzyLookupFactory</str>
<str name="storeDir">suggester_fuzzy_dir</str>
<str name="dictionaryImpl">DocumentDictionaryFactory</str>
<str name="field">autocomplete</str>
<str name="suggestAnalyzerFieldType">autocomplete</str>
<str name="buildOnOptimize">true</str>
<str name="buildOnStartup">true</str>
<str name="buildOnCommit">false</str>
</lst>
</searchComponent>
<requestHandler name="/suggesthandler" class="solr.SearchHandler" startup="lazy">
<lst name="defaults">
<str name="suggest">true</str>
<str name="suggest.dictionary">suggest</str>
<str name="suggest.count">10</str>
</lst>
<arr name="components">
<str>suggest</str>
</arr>
</requestHandler>
I have 10M+ entries in my base. My goal is autosuggestion on the title attribute.
This setup should index twice my title. Indeed, my index size doubled when I reindexed with those settings.
I have indeed a folder suggester_fuzzy_dir who was created in my core data folder.
However when I startup solr, or launch the request /suggesthandler?suggest.build=true, this suggester_fuzzy_dir folder doesn't grow in size, it always contains 1 byte. However the leftover SSD storage space on my disk is reducing, I wasn't able to see from where.
After 45 minutes I usually get a java heap space out of memory error.
My disk size returns to normal.
I tried launching solr with option -memory=4096m to allocate more (My computer have 8go RAM). This still doesn't work although it should be enough ? Which makes me think the problem is somewhere else.
Edit : The error returned by solr in the console is as follow :
{
"error": {
"msg": "java.lang.OutOfMemoryError: Java heap space",
"trace": "java.lang.RuntimeException: java.lang.OutOfMemoryError:
Java heap space\n\tat org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:618)\n\tat org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:477)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:214)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)\n\tat org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)\n\tat org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:499)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)\n\tat org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: java.lang.OutOfMemoryError: Java heap space\n\tat org.apache.lucene.util.packed.Packed64.<init>(Packed64.java:73)\n\tat org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:1009)\n\tat org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:976)\n\tat org.apache.lucene.util.packed.GrowableWriter.<init>(GrowableWriter.java:46)\n\tat org.apache.lucene.util.packed.PagedGrowableWriter.newMutable(PagedGrowableWriter.java:58)\n\tat org.apache.lucene.util.packed.AbstractPagedMutable.fillPages(AbstractPagedMutable.java:60)\n\tat org.apache.lucene.util.packed.PagedGrowableWriter.<init>(PagedGrowableWriter.java:52)\n\tat org.apache.lucene.util.packed.PagedGrowableWriter.<init>(PagedGrowableWriter.java:45)\n\tat org.apache.lucene.util.fst.NodeHash.rehash(NodeHash.java:164)\n\tat org.apache.lucene.util.fst.NodeHash.add(NodeHash.java:133)\n\tat org.apache.lucene.util.fst.Builder.compileNode(Builder.java:215)\n\tat org.apache.lucene.util.fst.Builder.freezeTail(Builder.java:310)\n\tat org.apache.lucene.util.fst.Builder.add(Builder.java:417)\n\tat org.apache.lucene.search.suggest.analyzing.AnalyzingSuggester.build(AnalyzingSuggester.java:557)\n\tat org.apache.lucene.search.suggest.Lookup.build(Lookup.java:193)\n\tat org.apache.solr.spelling.suggest.SolrSuggester.build(SolrSuggester.java:162)\n\tat org.apache.solr.handler.component.SuggestComponent.prepare(SuggestComponent.java:179)\n\tat org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:251)\n\tat org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)\n\tat org.apache.solr.core.SolrCore.execute(SolrCore.java:2068)\n\tat org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:669)\n\tat org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:462)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:214)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)\n\tat org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)\n",
"code": 500
}
}

Answer 1

So I finally made it work by increasing the memory allocated to the Java Virtual Machine.
In sunspot.yml :
development:
solr:
hostname: localhost
port: 8982
log_level: INFO
path: /solr/development
memory: 6G # => This allocate 6g RAM to the JVM
It might have worked with a 4 go memory allocation, thought. I checked the build real-time and there was some memory usage peaks over 2, sometimes 3g.
My suggester_fuzzy_dir now weight 1.3 go, which is more logical.