Occasional, unpredictable exceptions with multi-threaded solrj queries - solr

Question

I have a multi-threaded application using solrj 4. There are a maximum of 25 threads. Each thread creates a connection using HttpSolrServer, and runs one query. Most of the time this works just fine. But occasionally I get the following exception:
Jan 10, 2013 9:29:07 AM org.apache.http.impl.client.DefaultRequestDirector tryConnect
INFO: I/O exception (java.net.NoRouteToHostException) caught when connecting to the target host: Cannot assign requested address
Jan 10, 2013 9:29:07 AM org.apache.http.impl.client.DefaultRequestDirector tryConnect
INFO: Retrying connect
Exception in thread "main" java.util.concurrent.ExecutionException: java.lang.NullPointerException
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
I wrote some code to retry the query, if it fails:
while(!querySuccess && queryAttempts<m_MaxQueryAttempts ){
try{
queryAttempts++;
rsp = m_Server.query( query );
querySuccess = true;
}catch(SolrServerException e){
querySuccess = false;
}
}
After one or more retries the query usually works. But sometimes it fails even after 100 retries. Either way, I'd like to understand what the cause of the problem is. Why does it work some of the time? Is it an issue with concurrent access to solr? Apart from this process, I only have one other process that it continually writing to the index using a single connection. The default server settings are below - so I don't think it's because of too many simultaneous connections.
INFO: Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
Any suggestions on how to diagnose this would be much appreciated.