OutOfMemory with Solr4 - solr

Question

I'm indexing the content I have and after upgrade my Solr instance to solr 4 I'm facing some OutOfMemories. The exception thrown is:
INFO org.apache.solr.update.UpdateHandler - start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
ERROR o.a.solr.servlet.SolrDispatchFilter - null:java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space
at org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:469)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:297)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:240)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:164)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:164)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:562)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:395)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:250)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:188)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:166)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:302)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.OutOfMemoryError: Java heap space
Is there some known bug or something I could test to get rid of it?
Within this upgrade two things changed:
solr version (from 3.4 to 4.0);
lucene match version (from LUCENE_34 to LUCENE_40).

Answer 1

Seems to be running out of memory when accessing logs, at a glance. That may not be particularly meaningful, with an 'Out of Memory' error, of course, but worth a shot, particularly after seeing this complaint regarding SOLR 4.0 logging. Particularly so if this is occuring during an index rebuild of some form, or heavy load of updates.
So try disabling the update log, which I believe can be done by commenting out:
<updateLog>
<str name="dir">${solr.data.dir:}</str>
</updateLog>
in solrconfig.xml.
EDIT:
Another (possibly better) approach to this, taking another glance at it, might be to commit more often. The growth of the update log seems to be directly related to having a lot of queued updates waiting for commit.
If you do not have autocommit enabled, you might want to try adding it in your config, something like:
<autoCommit>
<maxTime>15000</maxTime>
<openSearcher>false</openSearcher>
</autoCommit>
There's a good bit of related discussion and recommendation to be found on this thread.

Answer 2

I ran into the same problem today and after reading #femtoRgon's suggested thread, I changed the following in the solrconfig.xml
<autoCommit>
<maxTime>15000</maxTime>
<openSearcher>false</openSearcher>
</autoCommit>
to
<autoCommit>
<maxDocs>15000</maxDocs>
<openSearcher>false</openSearcher>
</autoCommit>
It no longer gives me that error. So it commits every 15,000 docs. Which in my case is frequent enough not to run into memory issues. In my MacBook Pro it took few minutes to index ~4m documents containing product information (so short documents).