Solr getting stopped abruptly - solr

I am using Solr-4.3.1 on Ubuntu machine and it is running on default jetty server. Recently, I have observed that it gets shut down abruptly. I cant find any exception or error or warning message in the logs. Following are the logs:
org.eclipse.jetty.server.Server; Graceful shutdown
SocketConnector#0.0.0.0:8983
INFO - 2013-10-16 15:05:15.091; org.eclipse.jetty.server.Server;
Graceful shutdown
o.e.j.w.WebAppContext{/solr,file:/home/abhijeet/project/demo/Demo/solr-4.3.1/example/solr-webapp/webapp/},/home/abhijeet/project/demo/Demo/solr-4.3.1/example/webapps/solr.war
INFO - 2013-10-16 15:05:16.093; org.apache.solr.core.CoreContainer;
Shutting down CoreContainer instance=2137222734
INFO - 2013-10-16 15:05:16.236; org.apache.solr.core.SolrCore;
[collection1] CLOSING SolrCore org.apache.solr.core.SolrCore#9f07597
INFO - 2013-10-16 15:05:16.244;
org.apache.solr.update.DirectUpdateHandler2; closing
DirectUpdateHandler2{commits=9,autocommit
maxTime=15000ms,autocommits=0,soft
autocommits=0,optimizes=1,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
INFO - 2013-10-16 15:05:16.245; org.apache.solr.update.SolrCoreState;
Closing SolrCoreState
INFO - 2013-10-16 15:05:16.246;
org.apache.solr.update.DefaultSolrCoreState; SolrCoreState ref count
has reached 0 - closing IndexWriter
INFO - 2013-10-16 15:05:16.246;
org.apache.solr.update.DefaultSolrCoreState; closing IndexWriter with
IndexWriterCloser
INFO - 2013-10-16 15:05:16.330; org.apache.solr.core.SolrCore;
[collection1] Closing main searcher on request.
INFO - 2013-10-16 15:05:16.332;
org.apache.solr.core.CachingDirectoryFactory; Closing
NRTCachingDirectoryFactory - 2 directories currently being tracked
INFO - 2013-10-16 15:05:16.352;
org.apache.solr.core.CachingDirectoryFactory; looking to close
/home/abhijeet/project/demo/Demo/solr-4.3.1/example/solr/collection1/data
[CachedDir<>]
INFO - 2013-10-16 15:05:16.352;
org.apache.solr.core.CachingDirectoryFactory; Closing directory:
/home/abhijeet/project/demo/Demo/solr-4.3.1/example/solr/collection1/data
INFO - 2013-10-16 15:05:16.352;
org.apache.solr.core.CachingDirectoryFactory; looking to close
/home/abhijeet/project/demo/Demo/solr-4.3.1/example/solr/collection1/data/index
[CachedDir<>]
INFO - 2013-10-16 15:05:16.353;
org.apache.solr.core.CachingDirectoryFactory; Closing directory:
/home/abhijeet/project/demo/Demo/solr-4.3.1/example/solr/collection1/data/index
INFO - 2013-10-16 15:05:16.354;
org.eclipse.jetty.server.handler.ContextHandler; stopped
o.e.j.w.WebAppContext{/solr,file:/home/abhijeet/project/demo/Demo/solr-4.3.1/example/solr-webapp/webapp/},/home/abhijeet/project/demo/Demo/solr-4.3.1/example/webapps/solr.war
INFO - 2013-10-16 15:05:16.588;
org.eclipse.jetty.util.thread.ShutdownThread; shutdown already
commenced

Related

Corrupted files with Camel Ftp component

I'm using apache camel to make a ftp client for downloading some files to some local directory. The program reads a xml file to get the name of the file that will be fetched from the ftp.The program seems to work except that the files downloaded are corrupted. Right now I'm trying to download some image files but the ones I get are 14.9Kb and corrupted, no error message shown.
This is my code:
main
public void main() throws FileNotFoundException {
BasicConfigurator.configure();
RutaFtp routeBuilder = new RutaFtp();
CamelContext ctx = new DefaultCamelContext();
try {
ctx.addRoutes(routeBuilder);
ctx.start();
Thread.sleep(10000);
ctx.stop();
}
catch (Exception e) {
e.printStackTrace();
}
}
camel route:
from("file:./?fileName=Datos.xml&noop=true")
.split(xpath("//Datos/imagen/text()"))
.setProperty("rutaArchivo", this.body())
.log(LoggingLevel.INFO, "imagen: ${body}")
.process(ExtraerNombre).to("direct:ftp").end();
from("direct:ftp")
.pollEnrich("ftp://"+user+"#"+ftp+"/?password="+password+"&recursive=true&passiveMode=true&fileName=${body}&delete="+borrado+"")
.to("file:C:/outputFolder?flatten=true").end();
}
I've tried using the streamDownload parameter but tha prevents the files to be downloaded (I don't know why)
.pollEnrich("ftp://"+user+"#"+ftp+"/?password="+password+"&recursive=true&passiveMode=true&streamDownload=true&fileName=${body}&delete="+borrado+"")
console log:
INFO | Apache Camel 2.15.1.redhat-621084 (CamelContext: camel-1) is
starting 0 [main] INFO org.apache.camel.impl.DefaultCamelContext -
Apache Camel 2.15.1.redhat-621084 (CamelContext: camel-1) is starting
INFO | JMX is enabled 10 [main] INFO
org.apache.camel.management.ManagedManagementStrategy - JMX is
enabled INFO | Loaded 185 type converters 208 [main] INFO
org.apache.camel.impl.converter.DefaultTypeConverter - Loaded 185
type converters INFO | AllowUseOriginalMessage is enabled. If access
to the original message is not needed, then its recommended to turn
this option off as it may improve performance. 395 [main] INFO
org.apache.camel.impl.DefaultCamelContext - AllowUseOriginalMessage
is enabled. If access to the original message is not needed, then its
recommended to turn this option off as it may improve performance.
INFO | StreamCaching is not in use. If using streams then its
recommended to enable stream caching. See more details at
http://camel.apache.org/stream-caching.html 395 [main] INFO
org.apache.camel.impl.DefaultCamelContext - StreamCaching is not in
use. If using streams then its recommended to enable stream caching.
See more details at http://camel.apache.org/stream-caching.html INFO
| Endpoint is configured with noop=true so forcing endpoint to be
idempotent as well 395 [main] INFO
org.apache.camel.component.file.FileEndpoint - Endpoint is configured
with noop=true so forcing endpoint to be idempotent as well INFO |
Using default memory based idempotent repository with cache max size:
1000 395 [main] INFO org.apache.camel.component.file.FileEndpoint -
Using default memory based idempotent repository with cache max size:
1000 INFO | Route: route1 started and consuming from:
Endpoint[file://./?fileName=Datos.xml&noop=true] 502 [main] INFO
org.apache.camel.impl.DefaultCamelContext - Route: route1 started and
consuming from: Endpoint[file://./?fileName=Datos.xml&noop=true] INFO
| Route: route2 started and consuming from: Endpoint[direct://ftp] 504
[main] INFO org.apache.camel.impl.DefaultCamelContext - Route: route2
started and consuming from: Endpoint[direct://ftp] INFO | Total 2
routes, of which 2 is started. 504 [main] INFO
org.apache.camel.impl.DefaultCamelContext - Total 2 routes, of which
2 is started. INFO | Apache Camel 2.15.1.redhat-621084 (CamelContext:
camel-1) started in 0.504 seconds 507 [main] INFO
org.apache.camel.impl.DefaultCamelContext - Apache Camel
2.15.1.redhat-621084 (CamelContext: camel-1) started in 0.504 seconds INFO | Created default XPathFactory
com.sun.org.apache.xpath.internal.jaxp.XPathFactoryImpl#5434283f 1533
[Camel (camel-1) thread #0 - file://./] INFO
org.apache.camel.builder.xml.XPathBuilder - Created default
XPathFactory
com.sun.org.apache.xpath.internal.jaxp.XPathFactoryImpl#5434283f INFO
| imagen: ftp://190.0.56.190:8021/pruebasumman/conductor/71708375.jpg
1635 [Camel (camel-1) thread #0 - file://./] INFO route1 - imagen:
ftp://190.0.56.190:8021/pruebasumman/conductor/71708375.jpg INFO |
Apache Camel 2.15.1.redhat-621084 (CamelContext: camel-1) is shutting
down 10521 [main] INFO org.apache.camel.impl.DefaultCamelContext -
Apache Camel 2.15.1.redhat-621084 (CamelContext: camel-1) is shutting
down INFO | Starting to graceful shutdown 2 routes (timeout 300
seconds) 10524 [main] INFO
org.apache.camel.impl.DefaultShutdownStrategy - Starting to graceful
shutdown 2 routes (timeout 300 seconds) INFO | Waiting as there are
still 3 inflight and pending exchanges to complete, timeout in 300
seconds. 10524 [Camel (camel-1) thread #2 - ShutdownTask] INFO
org.apache.camel.impl.DefaultShutdownStrategy - Waiting as there are
still 3 inflight and pending exchanges to complete, timeout in 300
seconds. INFO | Waiting as there are still 3 inflight and pending
exchanges to complete, timeout in 299 seconds. 11525 [Camel (camel-1)
thread #2 - ShutdownTask] INFO
org.apache.camel.impl.DefaultShutdownStrategy - Waiting as there are
still 3 inflight and pending exchanges to complete, timeout in 299
seconds. INFO | Waiting as there are still 3 inflight and pending
exchanges to complete, timeout in 298 seconds. 12528 [Camel (camel-1)
thread #2 - ShutdownTask] INFO
org.apache.camel.impl.DefaultShutdownStrategy - Waiting as there are
still 3 inflight and pending exchanges to complete, timeout in 298
seconds. INFO | Waiting as there are still 3 inflight and pending
exchanges to complete, timeout in 297 seconds. 13529 [Camel (camel-1)
thread #2 - ShutdownTask] INFO
org.apache.camel.impl.DefaultShutdownStrategy - Waiting as there are
still 3 inflight and pending exchanges to complete, timeout in 297
seconds. INFO | Waiting as there are still 3 inflight and pending
exchanges to complete, timeout in 296 seconds. 14540 [Camel (camel-1)
thread #2 - ShutdownTask] INFO
org.apache.camel.impl.DefaultShutdownStrategy - Waiting as there are
still 3 inflight and pending exchanges to complete, timeout in 296
seconds. INFO | Waiting as there are still 3 inflight and pending
exchanges to complete, timeout in 295 seconds. 15555 [Camel (camel-1)
thread #2 - ShutdownTask] INFO
org.apache.camel.impl.DefaultShutdownStrategy - Waiting as there are
still 3 inflight and pending exchanges to complete, timeout in 295
seconds. INFO | Waiting as there are still 3 inflight and pending
exchanges to complete, timeout in 294 seconds. 16568 [Camel (camel-1)
thread #2 - ShutdownTask] INFO
org.apache.camel.impl.DefaultShutdownStrategy - Waiting as there are
still 3 inflight and pending exchanges to complete, timeout in 294
seconds. INFO | Waiting as there are still 3 inflight and pending
exchanges to complete, timeout in 293 seconds. 17569 [Camel (camel-1)
thread #2 - ShutdownTask] INFO
org.apache.camel.impl.DefaultShutdownStrategy - Waiting as there are
still 3 inflight and pending exchanges to complete, timeout in 293
seconds. INFO | Waiting as there are still 3 inflight and pending
exchanges to complete, timeout in 292 seconds. 18574 [Camel (camel-1)
thread #2 - ShutdownTask] INFO
org.apache.camel.impl.DefaultShutdownStrategy - Waiting as there are
still 3 inflight and pending exchanges to complete, timeout in 292
seconds.
Thanks in advance.
Download image file in binary mode
By default, Camel FTP is downloading file by ASCII mode.
Add binary=true into your ftp route will turn from ASCII mode to binary mode

hbase-indexer auto exit when data load frequently

Recently my team is using hbase-indexer on CDH for indexing hbase table column to solr . When we deploy hbase-indexer server (which is called Key-Value Store Indexer) and begin testing. We found a situation that when we put data to hbase (We are using apache phoenix, a SQL layer above hbase) frequently, the hbase-indexer process will auto exit.We have checked the log and found ERROR log of Zookeeper Session Expired like this :
2016-04-18 12:17:50,340 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 31481ms for sessionid 0x2541e69d8a2001a, closing socket connection and attempting reconnect
2016-04-18 12:17:50,446 WARN com.ngdata.hbaseindexer.util.zookeeper.StateWatchingZooKeeper: Disconnected from ZooKeeper
2016-04-18 12:17:51,202 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server slave1/192.168.27.166:2181. Will not attempt to authenticate using SASL (unknown error)
2016-04-18 12:17:51,204 INFO org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /192.168.27.167:59529, server: slave1/192.168.27.166:2181
2016-04-18 12:17:51,211 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x2541e69d8a2001a has expired, closing socket connection
2016-04-18 12:17:51,211 ERROR com.ngdata.hbaseindexer.util.zookeeper.StateWatchingZooKeeper: ZooKeeper session expired, shutting down.
2016-04-18 12:17:51,228 INFO org.mortbay.log: Stopped SelectChannelConnector#0.0.0.0:11060
2016-04-18 12:17:51,336 INFO com.ngdata.hbaseindexer.supervisor.IndexerSupervisor: IndexerWorker.EventWorker interrupted.
2016-04-18 12:17:51,448 INFO org.apache.zookeeper.ZooKeeper: Session: 0x2541e69d8a20020 closed
2016-04-18 12:17:51,448 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2016-04-18 12:17:51,462 INFO org.apache.hadoop.hbase.ipc.RpcServer: Stopping server on 44594
2016-04-18 12:17:51,463 INFO org.apache.hadoop.hbase.ipc.RpcServer: RpcServer.listener,port=44594: stopping
2016-04-18 12:17:51,473 INFO org.apache.hadoop.hbase.ipc.RpcServer: RpcServer.responder: stopped
2016-04-18 12:17:51,473 INFO org.apache.hadoop.hbase.ipc.RpcServer: RpcServer.responder: stopping
2016-04-18 12:17:51,488 ERROR com.ngdata.sep.util.io.Closer: Do not know how to close object of type com.ngdata.hbaseindexer.morphline.MorphlineResultToSolrMapper
2016-04-18 12:17:51,488 ERROR com.ngdata.sep.util.io.Closer: Do not know how to close object of type com.ngdata.hbaseindexer.uniquekey.StringUniqueKeyFormatter
2016-04-18 12:17:51,514 INFO org.apache.zookeeper.ZooKeeper: Session: 0x2541e69d8a2001f closed
2016-04-18 12:17:51,515 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2016-04-18 12:17:51,515 INFO org.apache.hadoop.hbase.ipc.RpcServer: Stopping server on 47364
2016-04-18 12:17:51,516 INFO org.apache.hadoop.hbase.ipc.RpcServer: RpcServer.listener,port=47364: stopping
2016-04-18 12:17:51,518 ERROR com.ngdata.sep.util.io.Closer: Do not know how to close object of type com.ngdata.hbaseindexer.morphline.MorphlineResultToSolrMapper
2016-04-18 12:17:51,518 INFO org.apache.hadoop.hbase.ipc.RpcServer: RpcServer.responder: stopped
2016-04-18 12:17:51,519 INFO org.apache.hadoop.hbase.ipc.RpcServer: RpcServer.responder: stopping
2016-04-18 12:17:51,519 ERROR com.ngdata.sep.util.io.Closer: Do not know how to close object of type com.ngdata.hbaseindexer.uniquekey.StringUniqueKeyFormatter
2016-04-18 12:17:51,527 INFO org.apache.zookeeper.ZooKeeper: Session: 0x2541e69d8a2001e closed
2016-04-18 12:17:51,527 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2016-04-18 12:17:51,528 INFO org.apache.hadoop.hbase.ipc.RpcServer: Stopping server on 49605
2016-04-18 12:17:51,528 INFO org.apache.hadoop.hbase.ipc.RpcServer: RpcServer.listener,port=49605: stopping
2016-04-18 12:17:51,530 INFO org.apache.hadoop.hbase.ipc.RpcServer: RpcServer.responder: stopped
2016-04-18 12:17:51,530 INFO org.apache.hadoop.hbase.ipc.RpcServer: RpcServer.responder: stopping
2016-04-18 12:17:51,531 ERROR com.ngdata.sep.util.io.Closer: Do not know how to close object of type com.ngdata.hbaseindexer.morphline.MorphlineResultToSolrMapper
2016-04-18 12:17:51,531 ERROR com.ngdata.sep.util.io.Closer: Do not know how to close object of type com.ngdata.hbaseindexer.uniquekey.StringUniqueKeyFormatter
2016-04-18 12:17:51,539 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2016-04-18 12:17:51,539 INFO org.apache.zookeeper.ZooKeeper: Session: 0x2541e69d8a2001c closed
2016-04-18 12:17:51,540 INFO org.apache.hadoop.hbase.ipc.RpcServer: Stopping server on 39464
2016-04-18 12:17:51,540 INFO org.apache.hadoop.hbase.ipc.RpcServer: RpcServer.listener,port=39464: stopping
2016-04-18 12:17:51,546 INFO org.apache.hadoop.hbase.ipc.RpcServer: RpcServer.responder: stopped
2016-04-18 12:17:51,547 INFO org.apache.hadoop.hbase.ipc.RpcServer: RpcServer.responder: stopping
2016-04-18 12:17:51,547 ERROR com.ngdata.sep.util.io.Closer: Do not know how to close object of type com.ngdata.hbaseindexer.morphline.MorphlineResultToSolrMapper
2016-04-18 12:17:51,547 ERROR com.ngdata.sep.util.io.Closer: Do not know how to close object of type com.ngdata.hbaseindexer.uniquekey.StringUniqueKeyFormatter
The software env is :
CDH5.4
HBase1.0
Phoenix4.6
Hbase-Indexer (hbase-solr-1.5-cdh5.4.2)
The java heapsize of hbase-indexer is configured to 1GB .
Is there anyone who meet this situation ?
All right, It due to the bad network of our cluster to make hbase-indexer zookeeper session timeout. And finally it cause hbase-indexer process to be shutdown automatically.

Solr suddenly shutsdown gracefully

I searched a lot for this problem but I can't find many resources concerning this issue.
We are running a set-up of SolrCloud on 2 FreeBsd servers. Every night at exactly 12:00 our Solr servers are somehow being shutdown gracefully internally by solr. Strange is that both solr-servers are being restarted.
I don't have any clu what the cause is, it seems that Jetty is the possible cause with the setting "stopAtShutdown" and "gracefulShutdown" but I am not sure what the reason is for restarting solr.
Some log lines:
INFO - 2015-11-24 00:00:01.056; org.eclipse.jetty.server.Server; Graceful shutdown SocketConnector#0.0.0.0:8983
INFO - 2015-11-24 00:00:01.057; org.eclipse.jetty.server.Server; Graceful shutdown o.e.j.w.WebAppContext{/solr,file:/usr/local/share/examples/apache-solr/solr-webapp/webapp/},/usr/local/share/examples/apa
che-solr/webapps/solr.war
INFO - 2015-11-24 00:00:02.064; org.apache.solr.core.CoreContainer; Shutting down CoreContainer instance=568196821
WARN - 2015-11-24 00:00:02.065; org.apache.solr.cloud.RecoveryStrategy; Stopping recovery for core=X_shard1_replica2 coreNodeName=core_node1
WARN - 2015-11-24 00:00:02.065; org.apache.solr.cloud.RecoveryStrategy; Stopping recovery for core=Y_shard1_replica1 coreNodeName=core_node1
INFO - 2015-11-24 00:00:02.065; org.apache.solr.cloud.ZkController; publishing core=Z_shard1_replica1 state=down collection=Z
INFO - 2015-11-24 00:00:02.073; org.apache.solr.cloud.ZkController; publishing core=Zoetermeer_Openbaar_shard1_replica2 state=down collection=Zoetermeer_Openbaar
INFO - 2015-11-24 00:00:02.102; org.apache.solr.cloud.ZkController; publishing core=X_shard1_replica2 state=down collection=X
INFO - 2015-11-24 00:00:02.110; org.apache.solr.cloud.ZkController; publishing core=Y_shard1_replica1 state=down collection=Y
INFO - 2015-11-24 00:00:02.125; org.apache.solr.core.SolrCore; [Z_shard1_replica1] CLOSING SolrCore org.apache.solr.core.SolrCore#3e6f43bd
INFO - 2015-11-24 00:00:02.125; org.apache.solr.update.DirectUpdateHandler2; closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,
adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=0}
INFO - 2015-11-24 00:00:02.126; org.apache.solr.update.SolrCoreState; Closing SolrCoreState
INFO - 2015-11-24 00:00:02.126; org.apache.solr.update.DefaultSolrCoreState; SolrCoreState ref count has reached 0 - closing IndexWriter
INFO - 2015-11-24 00:00:02.126; org.apache.solr.update.DefaultSolrCoreState; closing IndexWriter with IndexWriterCloser
INFO - 2015-11-24 00:00:02.129; org.apache.solr.core.SolrCore; [Z_shard1_replica1] Closing main searcher on request.
INFO - 2015-11-24 00:00:02.130; org.apache.solr.core.CachingDirectoryFactory; Closing StandardDirectoryFactory - 2 directories currently being tracked
INFO - 2015-11-24 00:00:02.132; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating
... (live nodes size: 2)
INFO - 2015-11-24 00:00:02.133; org.apache.solr.core.CachingDirectoryFactory; looking to close /data/solr/Z_shard1_replica1/data/index [CachedDir<<refCount=0;path=/data/solr/Z_shard1_replica1/data/index;done=false>>]
INFO - 2015-11-24 00:00:02.133; org.apache.solr.core.CachingDirectoryFactory; Closing directory: /data/solr/Z_shard1_replica1/data/index
INFO - 2015-11-24 00:00:02.134; org.apache.solr.core.CachingDirectoryFactory; looking to close /data/solr/Z_shard1_replica1/data [CachedDir<<refCount=0;path=/data/solr/Z_shard1_replica1/data;done=false>>]
INFO - 2015-11-24 00:00:02.134; org.apache.solr.core.CachingDirectoryFactory; Closing directory: /data/solr/Z_shard1_replica1/data
INFO - 2015-11-24 00:00:02.134; org.apache.solr.core.SolrCore; [X_Openbaar_shard1_replica2] CLOSING SolrCore org.apache.solr.core.SolrCore#78
Does somebody have an idea what is going on?

Loading solr configs in Cloudera SolrCloud

We try to import our data into SolrCloud using MapReduce batch indexing. We face a problem at the reduce phase, that solr.xml cannot be found. We create a 'twitter' collection but looking at the logs, after it failed to load in solr.xml, it uses the default one and tries to create 'collection1' (failed) and 'core1' (success) SolrCore. I'm not sure if we need to create our own solr.xml and where to put it (we try to put it at several places but it seems not to load in). Below is the log:
2022 [main] INFO org.apache.solr.hadoop.HeartBeater - Heart beat reporting class is org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
2025 [main] INFO org.apache.solr.hadoop.SolrRecordWriter - Using this unpacked directory as solr home: /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip
2025 [main] INFO org.apache.solr.hadoop.SolrRecordWriter - Creating embedded Solr server with solrHomeDir: /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip, fs: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-1828461666_1, ugi=nguyen (auth:SIMPLE)]], outputShardDir: hdfs://master.hadoop:8020/user/nguyen/twitter/outdir/reducers/_temporary/_attempt_201311191613_0320_r_000014_0/part-r-00014
2029 [Thread-64] INFO org.apache.solr.hadoop.HeartBeater - HeartBeat thread running
2030 [Thread-64] INFO org.apache.solr.hadoop.HeartBeater - Issuing heart beat for 1 threads
2083 [main] INFO org.apache.solr.core.SolrResourceLoader - new SolrResourceLoader for directory: '/data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/'
2259 [main] INFO org.apache.solr.hadoop.SolrRecordWriter - Constructed instance information solr.home /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip (/data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip), instance dir /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/, conf dir /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/conf/, writing index to solr.data.dir hdfs://master.hadoop:8020/user/nguyen/twitter/outdir/reducers/_temporary/_attempt_201311191613_0320_r_000014_0/part-r-00014/data, with permdir hdfs://master.hadoop:8020/user/nguyen/twitter/outdir/reducers/_temporary/_attempt_201311191613_0320_r_000014_0/part-r-00014
2266 [main] INFO org.apache.solr.core.ConfigSolr - Loading container configuration from /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/solr.xml
2267 [main] INFO org.apache.solr.core.ConfigSolr - /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/solr.xml does not exist, using default configuration
2505 [main] INFO org.apache.solr.core.CoreContainer - New CoreContainer 696103669
2505 [main] INFO org.apache.solr.core.CoreContainer - Loading cores into CoreContainer [instanceDir=/data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/]
2515 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory - Setting socketTimeout to: 0
2515 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory - Setting urlScheme to: http://
2515 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory - Setting connTimeout to: 0
2515 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory - Setting maxConnectionsPerHost to: 20
2516 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory - Setting corePoolSize to: 0
2516 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory - Setting maximumPoolSize to: 2147483647
2516 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory - Setting maxThreadIdleTime to: 5
2516 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory - Setting sizeOfQueue to: -1
2516 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory - Setting fairnessPolicy to: false
2527 [main] INFO org.apache.solr.client.solrj.impl.HttpClientUtil - Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
2648 [main] INFO org.apache.solr.logging.LogWatcher - Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
2676 [coreLoadExecutor-3-thread-1] INFO org.apache.solr.core.CoreContainer - Creating SolrCore 'collection1' using instanceDir: /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/collection1
2677 [coreLoadExecutor-3-thread-1] INFO org.apache.solr.core.SolrResourceLoader - new SolrResourceLoader for directory: '/data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/collection1/'
2691 [coreLoadExecutor-3-thread-1] ERROR org.apache.solr.core.CoreContainer - Failed to load file /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/collection1/solrconfig.xml
2693 [coreLoadExecutor-3-thread-1] ERROR org.apache.solr.core.CoreContainer - Unable to create core: collection1
org.apache.solr.common.SolrException: Could not load config for solrconfig.xml
at org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:596)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:661)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:368)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:360)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.io.IOException: Can't find resource 'solrconfig.xml' in classpath or '/data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/collection1/conf/', cwd=/data/05/mapred/local/taskTracker/nguyen/jobcache/job_201311191613_0320/attempt_201311191613_0320_r_000014_0/work
at org.apache.solr.core.SolrResourceLoader.openResource(SolrResourceLoader.java:322)
at org.apache.solr.core.SolrResourceLoader.openConfig(SolrResourceLoader.java:287)
at org.apache.solr.core.Config.<init>(Config.java:116)
at org.apache.solr.core.Config.<init>(Config.java:86)
at org.apache.solr.core.SolrConfig.<init>(SolrConfig.java:120)
at org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:593)
... 11 more
2695 [coreLoadExecutor-3-thread-1] ERROR org.apache.solr.core.CoreContainer - null:org.apache.solr.common.SolrException: Unable to create core: collection1
at org.apache.solr.core.CoreContainer.recordAndThrow(CoreContainer.java:1158)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:670)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:368)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:360)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: org.apache.solr.common.SolrException: Could not load config for solrconfig.xml
at org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:596)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:661)
... 10 more
Caused by: java.io.IOException: Can't find resource 'solrconfig.xml' in classpath or '/data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/collection1/conf/', cwd=/data/05/mapred/local/taskTracker/nguyen/jobcache/job_201311191613_0320/attempt_201311191613_0320_r_000014_0/work
at org.apache.solr.core.SolrResourceLoader.openResource(SolrResourceLoader.java:322)
at org.apache.solr.core.SolrResourceLoader.openConfig(SolrResourceLoader.java:287)
at org.apache.solr.core.Config.<init>(Config.java:116)
at org.apache.solr.core.Config.<init>(Config.java:86)
at org.apache.solr.core.SolrConfig.<init>(SolrConfig.java:120)
at org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:593)
... 11 more
2697 [main] INFO org.apache.solr.core.CoreContainer - Creating SolrCore 'core1' using instanceDir: /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip
2697 [main] INFO org.apache.solr.core.SolrResourceLoader - new SolrResourceLoader for directory: '/data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/'
2751 [main] INFO org.apache.solr.core.SolrConfig - Adding specified lib dirs to ClassLoader
2752 [main] WARN org.apache.solr.core.SolrResourceLoader - Can't find (or read) directory to add to classloader: ../../../contrib/extraction/lib (resolved as: /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/../../../contrib/extraction/lib).
2752 [main] WARN org.apache.solr.core.SolrResourceLoader - Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/../../../dist).
2752 [main] WARN org.apache.solr.core.SolrResourceLoader - Can't find (or read) directory to add to classloader: ../../../contrib/clustering/lib/ (resolved as: /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/../../../contrib/clustering/lib).
2753 [main] WARN org.apache.solr.core.SolrResourceLoader - Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/../../../dist).
2753 [main] WARN org.apache.solr.core.SolrResourceLoader - Can't find (or read) directory to add to classloader: ../../../contrib/langid/lib/ (resolved as: /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/../../../contrib/langid/lib).
2753 [main] WARN org.apache.solr.core.SolrResourceLoader - Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/../../../dist).
2753 [main] WARN org.apache.solr.core.SolrResourceLoader - Can't find (or read) directory to add to classloader: ../../../contrib/velocity/lib (resolved as: /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/../../../contrib/velocity/lib).
2753 [main] WARN org.apache.solr.core.SolrResourceLoader - Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/../../../dist).
2785 [main] INFO org.apache.solr.update.SolrIndexConfig - IndexWriter infoStream solr logging is enabled
2790 [main] INFO org.apache.solr.core.SolrConfig - Using Lucene MatchVersion: LUCENE_44
2869 [main] INFO org.apache.solr.core.Config - Loaded SolrConfig: solrconfig.xml
2879 [main] INFO org.apache.solr.schema.IndexSchema - Reading Solr Schema from schema.xml
2937 [main] INFO org.apache.solr.schema.IndexSchema - [core1] Schema name=twitter
3352 [main] INFO org.apache.solr.schema.IndexSchema - unique key field: id
3471 [main] INFO org.apache.solr.schema.FileExchangeRateProvider - Reloading exchange rates from file currency.xml
3478 [main] INFO org.apache.solr.schema.FileExchangeRateProvider - Reloading exchange rates from file currency.xml
3635 [main] INFO org.apache.solr.core.HdfsDirectoryFactory - Solr Kerberos Authentication disabled
3636 [main] INFO org.apache.solr.core.JmxMonitoredMap - No JMX servers found, not exposing Solr information with JMX.
3652 [main] INFO org.apache.solr.core.HdfsDirectoryFactory - creating directory factory for path hdfs://master.hadoop:8020/user/nguyen/twitter/outdir/reducers/_temporary/_attempt_201311191613_0320_r_000014_0/part-r-00014/data
3686 [main] INFO org.apache.solr.core.CachingDirectoryFactory - return new directory for hdfs://master.hadoop:8020/user/nguyen/twitter/outdir/reducers/_temporary/_attempt_201311191613_0320_r_000014_0/part-r-00014/data
3711 [main] WARN org.apache.solr.core.SolrCore - [core1] Solr index directory 'hdfs:/master.hadoop:8020/user/nguyen/twitter/outdir/reducers/_temporary/_attempt_201311191613_0320_r_000014_0/part-r-00014/data/index' doesn't exist. Creating new index...
3719 [main] INFO org.apache.solr.core.HdfsDirectoryFactory - creating directory factory for path hdfs://master.hadoop:8020/user/nguyen/twitter/outdir/reducers/_temporary/_attempt_201311191613_0320_r_000014_0/part-r-00014/data/index
3719 [main] INFO org.apache.solr.core.HdfsDirectoryFactory - Number of slabs of block cache [1] with direct memory allocation set to [true]
3720 [main] INFO org.apache.solr.core.HdfsDirectoryFactory - Block cache target memory usage, slab size of [134217728] will allocate [1] slabs and use ~[134217728] bytes
3721 [main] INFO org.apache.solr.store.blockcache.BufferStore - Initializing the 1024 buffers with [8192] buffers.
3740 [main] INFO org.apache.solr.store.blockcache.BufferStore - Initializing the 8192 buffers with [8192] buffers.
3891 [main] INFO org.apache.solr.core.CachingDirectoryFactory - return new directory for hdfs://master.hadoop:8020/user/nguyen/twitter/outdir/reducers/_temporary/_attempt_201311191613_0320_r_000014_0/part-r-00014/data/index
3988 [main] INFO org.apache.solr.update.LoggingInfoStream - [IFD][main]: init: current segments file is "null"; deletionPolicy=org.apache.solr.core.IndexDeletionPolicyWrapper#65b01d5d
3992 [main] INFO org.apache.solr.update.LoggingInfoStream - [IFD][main]: now checkpoint "" [0 segments ; isCommit = false]
3992 [main] INFO org.apache.solr.update.LoggingInfoStream - [IFD][main]: 0 msec to checkpoint
3992 [main] INFO org.apache.solr.update.LoggingInfoStream - [IW][main]: init: create=true
3992 [main] INFO org.apache.solr.update.LoggingInfoStream - [IW][main]:
dir=NRTCachingDirectory(org.apache.solr.store.hdfs.HdfsDirectory#17e5a6d8 lockFactory=org.apache.solr.store.hdfs.HdfsLockFactory#7f117668; maxCacheMB=192.0 maxMergeSizeMB=16.0)
solr looks for solr.home parameter and searchs solrConfig.xml file there. if there is none it tries to load default configuration.
it looks like your solr home is
/data/06/mapred/local/taskTracker/distcache/3866561797898787678_-1754062477_512745567/master.hadoop/tmp/9501daf9-5011-4665-bae3-d5af1c8bcd62.solr.zip/collection1/
check that folder for solrconfig.xml file
if there is none, copy one from example directory of solr
if there is one, match the file/folder permissions with the server instance

zookeeper does not run?

I wanted to run a solr cloud with solr 4.3.0.
(I am using aws ubuntu-12.04-lts micro instances)
So I followed this toturial:
which basically says, start the zookeeper and connect the solr instances to it.
Here's how I start the zookeeper.
First I copied the config like described in the tutorial
sudo cp zookeeper-3.4.5/conf/zoo_sample.cfg zookeeper-3.4.5/conf/zoo.cfg
Then I started the zookeeper
ubuntu#ip-10-48-159-36:/opt$ sudo zookeeper-3.4.5/bin/zkServer.sh start
JMX enabled by default
Using config: /opt/zookeeper-3.4.5/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
Looks fine so far.
I checked the status:
ubuntu#ip-10-48-159-36:/opt$ sudo zookeeper-3.4.5/bin/zkServer.sh status
JMX enabled by default
Using config: /opt/zookeeper-3.4.5/bin/../conf/zoo.cfg
Error contacting service. It is probably not running.
Which seems a bit weird already.
If I try to connect with the client (remote as well as local), its seems to work
ubuntu#ip-10-234-223-69:/opt$ zookeeper-3.4.5/bin/zkCli.sh -server ec2-54-247-144-120.eu-west-1.compute.amazonaws.com:2181
Connecting to ec2-54-247-144-120.eu-west-1.compute.amazonaws.com:2181
2013-06-07 11:07:01,996 [myid:] - INFO [main:Environment#100] - Client environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
2013-06-07 11:07:02,000 [myid:] - INFO [main:Environment#100] - Client environment:host.name=ip-10-234-223-69.eu-west-1.compute.internal
2013-06-07 11:07:02,000 [myid:] - INFO [main:Environment#100] - Client environment:java.version=1.6.0_27
2013-06-07 11:07:02,002 [myid:] - INFO [main:Environment#100] - Client environment:java.vendor=Sun Microsystems Inc.
2013-06-07 11:07:02,003 [myid:] - INFO [main:Environment#100] - Client environment:java.home=/usr/lib/jvm/java-6-openjdk-amd64/jre
2013-06-07 11:07:02,003 [myid:] - INFO [main:Environment#100] - Client environment:java.class.path=/opt/zookeeper-3.4.5/bin/../build/classes:/opt/zookeeper-3.4.5/bin/../build/lib/*.jar:/opt/zookeeper-3.4.5/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/zookeeper-3.4.5/bin/../lib/slf4j-api-1.6.1.jar:/opt/zookeeper-3.4.5/bin/../lib/netty-3.2.2.Final.jar:/opt/zookeeper-3.4.5/bin/../lib/log4j-1.2.15.jar:/opt/zookeeper-3.4.5/bin/../lib/jline-0.9.94.jar:/opt/zookeeper-3.4.5/bin/../zookeeper-3.4.5.jar:/opt/zookeeper-3.4.5/bin/../src/java/lib/*.jar:/opt/zookeeper-3.4.5/bin/../conf:
2013-06-07 11:07:02,004 [myid:] - INFO [main:Environment#100] - Client environment:java.library.path=/usr/lib/jvm/java-6-openjdk-amd64/jre/lib/amd64/server:/usr/lib/jvm/java-6-openjdk-amd64/jre/lib/amd64:/usr/lib/jvm/java-6-openjdk-amd64/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
2013-06-07 11:07:02,008 [myid:] - INFO [main:Environment#100] - Client environment:java.io.tmpdir=/tmp
2013-06-07 11:07:02,009 [myid:] - INFO [main:Environment#100] - Client environment:java.compiler=<NA>
2013-06-07 11:07:02,018 [myid:] - INFO [main:Environment#100] - Client environment:os.name=Linux
2013-06-07 11:07:02,019 [myid:] - INFO [main:Environment#100] - Client environment:os.arch=amd64
2013-06-07 11:07:02,019 [myid:] - INFO [main:Environment#100] - Client environment:os.version=3.2.0-40-virtual
2013-06-07 11:07:02,020 [myid:] - INFO [main:Environment#100] - Client environment:user.name=ubuntu
2013-06-07 11:07:02,020 [myid:] - INFO [main:Environment#100] - Client environment:user.home=/home/ubuntu
2013-06-07 11:07:02,021 [myid:] - INFO [main:Environment#100] - Client environment:user.dir=/opt
2013-06-07 11:07:02,029 [myid:] - INFO [main:ZooKeeper#438] - Initiating client connection, connectString=ec2-54-247-144-120.eu-west-1.compute.amazonaws.com:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher#182d9c06
Welcome to ZooKeeper!
2013-06-07 11:07:02,074 [myid:] - INFO [main-SendThread(ip-10-48-159-36.eu-west-1.compute.internal:2181):ClientCnxn$SendThread#966] - Opening socket connection to server ip-10-48-159-36.eu-west-1.compute.internal/10.48.159.36:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
[zk: ec2-54-247-144-120.eu-west-1.compute.amazonaws.com:2181(CONNECTING) 0] 2013-06-07 11:07:32,100 [myid:] - INFO [main-SendThread(ip-10-48-159-36.eu-west-1.compute.internal:2181):ClientCnxn$SendThread#1083] - Client session timed out, have not heard from server in 30038ms for sessionid 0x0, closing socket connection and attempting reconnect
2013-06-07 11:07:33,204 [myid:] - INFO [main-SendThread(ip-10-48-159-36.eu-west-1.compute.internal:2181):ClientCnxn$SendThread#966] - Opening socket connection to server ip-10-48-159-36.eu-west-1.compute.internal/10.48.159.36:2181. Will not attempt to authenticate using SASL (unknown error)
Now I tried to connect a solr instance to it. In the web interface of tomcat7 it only tells me "503 - Server is shutting down", so I checked the solr logs
2013-06-07 11:16:36,065 [pool-2-thread-1] INFO org.apache.solr.servlet.SolrDispatchFilter . SolrDispatchFilter.init()
2013-06-07 11:16:36,100 [pool-2-thread-1] INFO org.apache.solr.core.SolrResourceLoader . Using JNDI solr.home: /opt/solr-4.3.0/example/solr
2013-06-07 11:16:36,132 [pool-2-thread-1] INFO org.apache.solr.core.CoreContainer . looking for solr config file: /opt/solr-4.3.0/example/solr/solr.xml
2013-06-07 11:16:36,138 [pool-2-thread-1] INFO org.apache.solr.core.CoreContainer . New CoreContainer 1285984216
2013-06-07 11:16:36,146 [pool-2-thread-1] INFO org.apache.solr.core.CoreContainer . Loading CoreContainer using Solr Home: '/opt/solr-4.3.0/example/solr/'
2013-06-07 11:16:36,152 [pool-2-thread-1] INFO org.apache.solr.core.SolrResourceLoader . new SolrResourceLoader for directory: '/opt/solr-4.3.0/example/solr/'
2013-06-07 11:16:36,567 [pool-2-thread-1] INFO org.apache.solr.handler.component.HttpShardHandlerFactory . Setting socketTimeout to: 0
2013-06-07 11:16:36,568 [pool-2-thread-1] INFO org.apache.solr.handler.component.HttpShardHandlerFactory . Setting urlScheme to: http://
2013-06-07 11:16:36,568 [pool-2-thread-1] INFO org.apache.solr.handler.component.HttpShardHandlerFactory . Setting connTimeout to: 0
2013-06-07 11:16:36,568 [pool-2-thread-1] INFO org.apache.solr.handler.component.HttpShardHandlerFactory . Setting maxConnectionsPerHost to: 20
2013-06-07 11:16:36,568 [pool-2-thread-1] INFO org.apache.solr.handler.component.HttpShardHandlerFactory . Setting corePoolSize to: 0
2013-06-07 11:16:36,568 [pool-2-thread-1] INFO org.apache.solr.handler.component.HttpShardHandlerFactory . Setting maximumPoolSize to: 2147483647
2013-06-07 11:16:36,568 [pool-2-thread-1] INFO org.apache.solr.handler.component.HttpShardHandlerFactory . Setting maxThreadIdleTime to: 5
2013-06-07 11:16:36,569 [pool-2-thread-1] INFO org.apache.solr.handler.component.HttpShardHandlerFactory . Setting sizeOfQueue to: -1
2013-06-07 11:16:36,569 [pool-2-thread-1] INFO org.apache.solr.handler.component.HttpShardHandlerFactory . Setting fairnessPolicy to: false
2013-06-07 11:16:36,578 [pool-2-thread-1] INFO org.apache.solr.client.solrj.impl.HttpClientUtil . Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
2013-06-07 11:16:36,879 [pool-2-thread-1] INFO org.apache.solr.core.CoreContainer . Registering Log Listener
2013-06-07 11:16:36,881 [pool-2-thread-1] INFO org.apache.solr.core.CoreContainer . Zookeeper client=ec2-54-247-144-120.eu-west-1.compute.amazonaws.com:2181
2013-06-07 11:16:36,888 [pool-2-thread-1] INFO org.apache.solr.client.solrj.impl.HttpClientUtil . Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=0&connTimeout=0
2013-06-07 11:16:37,040 [pool-2-thread-1] INFO org.apache.solr.common.cloud.ConnectionManager . Waiting for client to connect to ZooKeeper
2013-06-07 11:16:52,046 [pool-2-thread-1] ERROR org.apache.solr.servlet.SolrDispatchFilter . Could not start Solr. Check solr/home property and the logs
2013-06-07 11:16:52,103 [pool-2-thread-1] ERROR org.apache.solr.core.SolrCore . null:java.lang.RuntimeException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper ec2-54-247-144-120.eu-west-1.compute.amazonaws.com:2181 within 15000 ms
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:130)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:88)
at org.apache.solr.cloud.ZkController.<init>(ZkController.java:170)
at org.apache.solr.core.CoreContainer.initZooKeeper(CoreContainer.java:242)
at org.apache.solr.core.CoreContainer.load(CoreContainer.java:495)
at org.apache.solr.core.CoreContainer.load(CoreContainer.java:358)
at org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:326)
at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:124)
at org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:277)
at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:258)
at org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:382)
at org.apache.catalina.core.ApplicationFilterConfig.<init>(ApplicationFilterConfig.java:103)
at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4638)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5294)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:895)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:871)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:615)
at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:649)
at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1581)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper ec2-54-247-144-120.eu-west-1.compute.amazonaws.com:2181 within 15000 ms
at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:173)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:127)
... 25 more
2013-06-07 11:16:52,104 [pool-2-thread-1] INFO org.apache.solr.servlet.SolrDispatchFilter . SolrDispatchFilter.init() done
What does it tell me?
On the same instance I just connected with the client successfully... :(
So where is the problem?
[Edit:]
Instead of using amazons ec**.amazon.* address I used the network addresses 10.X.X.X for telling solr where the zookeeper is.
It seems to work.
You have your answer - Your ZooKeeper in inaccessible!
Check your firewall configuration.
You can also check it with
zkCli.sh -server localhost:2181
There must have been some sort of connectivity problem.
I see you have it resolved now.
Next time you run into a situation like this, you should log onto the box that is having problems connecting and use telnet to see if you can connect.
eg: from your solr box:
telnet ec2-54-247-144-120.eu-west-1.compute.amazonaws.com 2181
and then try from the zk box too. It should start to illuminate where your issues are.
That eliminates any application layer issues and will tell you quite reliably wether or not you can connect. It you can't connect, then it's almost always some sort of security issue - either a firewall running somewhere (try - $service iptables stop) or it will be an issue with security group configuration in amazon.
The last potential problem is network availability. Despite what people think, the network is NOT reliable and should never be considered so. Anyone working in SOA/distributed systems will know this well :)
http://aphyr.com/posts/288-the-network-is-reliable
"A team from the University of Toronto and Microsoft Research studied the behavior of network failures in several of Microsoft’s datacenters. They found an average failure rate of 5.2 devices per day and 40.8 links per day with a median time to repair of approximately five minutes (and up to one week). "
While setting up SolrCloud and ZooKeeper I also ran into the "Error contacting service. It is probably not running." issue. The reason was a typo in a file name that ZooKeeper needs. The correct file name is "myid". I wrote "myip" by mistake. After the renaming of the file and restarting ZooKeeper (./zkServer.sh restart), my issue was resolved.
try to stop your solr instance solr.shutdown() so that you can create new CloudSolrServer instance for each thread

Resources