Camel SSH keyPairProvider option - apache-camel

Can anyone suggest how to specify the keyPairProvider option with Camel SSH component?
I am trying to connect to a remote SSH server using Camel SSH component and KeyPairProvider option. The documentation is not clear on how to use this option, the syntax or the value we need to provide etc. I have tried giving the classname, full classpath, key file name etc. Nothing works. I always get the below error.
Caused by: java.lang.IllegalArgumentException: Could not find a suitable setter for property: keyPairProvider as there isn't a setter method with same type: java.lang.String nor type conversion possible: No type converter available to convert from type: java.lang.String to the required type: org.apache.sshd.common.KeyPairProvider with value file:com.example.tree.custom.CustomFileKeyPairProvider
at org.apache.camel.util.IntrospectionSupport.setProperty(IntrospectionSupport.java:555)
at org.apache.camel.util.IntrospectionSupport.setProperty(IntrospectionSupport.java:565)
at org.apache.camel.util.IntrospectionSupport.setProperties(IntrospectionSupport.java:453)
at org.apache.camel.util.EndpointHelper.setProperties(EndpointHelper.java:249)
at org.apache.camel.impl.DefaultComponent.setProperties(DefaultComponent.java:264)
at org.apache.camel.component.ssh.SshComponent.createEndpoint(SshComponent.java:47)
at org.apache.camel.impl.DefaultComponent.createEndpoint(DefaultComponent.java:120)
at org.apache.camel.impl.DefaultCamelContext.getEndpoint(DefaultCamelContext.java:508)
... 18 more
This is my route
from("direct:sshroute")
.to("ssh://sshuser1#10.115.136.176?keyPairProvider=com.example.tree.custom.CustomFileKeyPairProvider")
If I do not give keyPairProvider and change the route to use password, it works fine.
I also want to mention that if I create my own SSHComponent, set the CustomFileKeyPairProvider and add it to CamelContext (as given in the tests package), it is works fine too.
But just want to know if we use default Camel Component, how do we use this option.
Thanks
- Ravi

I have been trying to come up with the answer to this because I also need to be able to deliver files via the Camel SFTP route. Luckily, I have found what works.
Basically, what you need to do is present the Private key, not the Public key, to your Camel route. The assumption here is that you generated a public key to share with the remote SSH/SFTP server, delivered that public key to the remote site and, either you or the remote administrator, placed that file in the "authorized_keys" container on the remote site.
That being said, you construct your Camel SFTP route as such:
<to uri="sftp://username#remote_server_address/directoryname?privateKeyFile=/full/path/to/your/private_key/id_dsa&knownHostsFile=/full/path/to/your/known_hosts" />
With this address, I was able to send my file to my remote destination without logging on:
11/27 23:35:03 INFO [aCamelQueue]] SftpOperations - JSCH -> Connecting to 10.0.0.110 port 22
11/27 23:35:03 INFO [aCamelQueue]] SftpOperations - JSCH -> Connection established
11/27 23:35:03 INFO [aCamelQueue]] SftpOperations - JSCH -> Remote version string: SSH-2.0-OpenSSH_6.4
11/27 23:35:03 INFO [aCamelQueue]] SftpOperations - JSCH -> Local version string: SSH-2.0-JSCH-0.1.44
11/27 23:35:03 INFO [aCamelQueue]] SftpOperations - JSCH -> CheckCiphers: aes256-ctr,aes192-ctr,aes128-ctr,aes256-cbc,aes192-cbc,aes128-cbc,3des-ctr,arcfour,arcfour128,arcfour256
11/27 23:35:03 INFO [aCamelQueue]] SftpOperations - JSCH -> SSH_MSG_KEXINIT sent
11/27 23:35:03 INFO [aCamelQueue]] SftpOperations - JSCH -> SSH_MSG_KEXINIT received
11/27 23:35:03 INFO [aCamelQueue]] SftpOperations - JSCH -> kex: server->client aes128-ctr hmac-md5 none
11/27 23:35:03 INFO [aCamelQueue]] SftpOperations - JSCH -> kex: client->server aes128-ctr hmac-md5 none
11/27 23:35:03 INFO [aCamelQueue]] SftpOperations - JSCH -> SSH_MSG_KEXDH_INIT sent
11/27 23:35:03 INFO [aCamelQueue]] SftpOperations - JSCH -> expecting SSH_MSG_KEXDH_REPLY
11/27 23:35:03 INFO [aCamelQueue]] SftpOperations - JSCH -> ssh_rsa_verify: signature true
11/27 23:35:03 INFO [aCamelQueue]] SftpOperations - JSCH -> Host '10.0.0.110' is known and mathces the RSA host key
11/27 23:35:03 INFO [aCamelQueue]] SftpOperations - JSCH -> SSH_MSG_NEWKEYS sent
11/27 23:35:03 INFO [aCamelQueue]] SftpOperations - JSCH -> SSH_MSG_NEWKEYS received
11/27 23:35:03 INFO [aCamelQueue]] SftpOperations - JSCH -> SSH_MSG_SERVICE_REQUEST sent
11/27 23:35:03 INFO [aCamelQueue]] SftpOperations - JSCH -> SSH_MSG_SERVICE_ACCEPT received
11/27 23:35:03 INFO [aCamelQueue]] SftpOperations - JSCH -> Authentications that can continue: publickey,keyboard-interactive,password
11/27 23:35:03 INFO [aCamelQueue]] SftpOperations - JSCH -> Next authentication method: publickey
11/27 23:35:03 INFO [aCamelQueue]] SftpOperations - JSCH -> Authentication succeeded (publickey)
I go into a bit more detail in my findings over on my website but this is the main jist. Good luck!

if you're looking for SSH connection using key pair, you can do it with the certResource option, like
from("direct:sshroute")
.to("ssh://sshuser1#10.115.136.176?certResource=file:path/to/your/private_key)
it's said in the doc that you can prefix path with classpath:, file:, or http:, but I've only tried file:.

Related

Remote connection to [null] failed with java.net.NoRouteToHostException: No route to host in taskmanager

When I start my apache flink 1.10 taskmanager service in kubernetes(v1.15.2) cluster,it shows logs like this:
2020-05-01 08:34:55,847 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink#flink-jobmanager:6123/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink#flink-jobmanager:6123/user/resourcemanager..
2020-05-01 08:34:55,847 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.NoRouteToHostException: No route to host
2020-05-01 08:34:55,848 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink#flink-jobmanager:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink#flink-jobmanager:6123]] Caused by: [java.net.NoRouteToHostException: No route to host]
2020-05-01 08:35:08,874 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.NoRouteToHostException: No route to host
2020-05-01 08:35:08,877 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink#flink-jobmanager:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink#flink-jobmanager:6123]] Caused by: [java.net.NoRouteToHostException: No route to host]
2020-05-01 08:35:08,878 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink#flink-jobmanager:6123/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink#flink-jobmanager:6123/user/resourcemanager..
2020-05-01 08:35:21,907 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.NoRouteToHostException: No route to host
and the taskmanager could not registered success, and I logged into taskmanager and find out I could success ping jobmanager liket this:
flink#flink-taskmanager-54d85f57c7-nl9cf:~$ ping flink-jobmanager
PING flink-jobmanager.dabai-fat.svc.cluster.local (10.254.58.171) 56(84) bytes of data.
64 bytes from flink-jobmanager.dabai-fat.svc.cluster.local (10.254.58.171): icmp_seq=1 ttl=64 time=0.045 ms
64 bytes from flink-jobmanager.dabai-fat.svc.cluster.local (10.254.58.171): icmp_seq=2 ttl=64 time=0.076 ms
64 bytes from flink-jobmanager.dabai-fat.svc.cluster.local (10.254.58.171): icmp_seq=3 ttl=64 time=0.079 ms
so why this would happen and what should I do to fix it?
Try to install nmap in your kubernetes taskmanger's pod container:
apt-get udpate
apt-get install nmap -y
then scan the jobmanager and make sure the pod's expose port 6123 is accessable(in my case ,I found could not access the port 6123 from current pod).
nmap -T4 <your-jobmanager's-pod-ip>
Hope this help.

Cannot initialize database connection from spring boot application

I am able to successfully connect to my local database, by specifying following URL in application.properties file of my spring boot application:
application.properties
spring.datasource.url=jdbc:sqlserver://localhost:1433;databaseName=mantaDB;integratedSecurity=true
And here is my output:
2019-Aug-29 11:14:41.298 INFO com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Starting... -
2019-Aug-29 11:14:41.753 INFO com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Start completed. -
2019-Aug-29 11:14:41.914 INFO o.h.jpa.internal.util.LogHelper - HHH000204: Processing PersistenceUnitInfo [_ name: default_ ...] -
2019-Aug-29 11:14:42.102 INFO org.hibernate.Version - HHH000412: Hibernate Core {5.3.7.Final} -
2019-Aug-29 11:14:42.396 INFO o.h.annotations.common.Version - HCANN000001: Hibernate Commons Annotations {5.0.4.Final} -
2019-Aug-29 11:14:42.827 INFO org.hibernate.dialect.Dialect - HHH000400: Using dialect: org.hibernate.dialect.SQLServerDialect -
2019-Aug-29 11:14:44.157 INFO o.s.o.j.LocalContainerEntityManagerFactoryBean - Initialized JPA EntityManagerFactory for persistence unit 'default' -
Now, I want to change application to point to my dev db box:
application.properties
spring.datasource.url=jdbc:sqlserver://STUDENT12.site,28001;databaseName=mantaDB;integratedSecurity=true
But, I am getting following error:
2019-Aug-29 11:16:48.089 INFO com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Starting... -
2019-Aug-29 11:17:18.682 ERROR com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Exception during pool initialization. -
com.microsoft.sqlserver.jdbc.SQLServerException: The TCP/IP connection to the host STUDENT12.site,28001, port 1433 has failed. Error: "STUDENT12.site,28001. Verify the connection properties. Make sure that an instance of SQL Server is running on the host and accepting TCP/IP connections at the port. Make sure that TCP connections to the port are not blocked by a firewall.".
at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(SQLServerException.java:228)
at com.microsoft.sqlserver.jdbc.SQLServerException.ConvertConnectExceptionToSQLServerException(SQLServerException.java:285)
Clearly, I can get to the database.
Does anyone know what i am doing wrong?
The JDBC Url should look like:
jdbc:sqlserver://[serverName[\instanceName][:portNumber]][;property=value[;property=value]]
Building the Connection URL
And the error makes it clear that you are not connecting on port 28001:
connection to the host JD1LSTWLSLMC101.dcsr.site,28001, port 1433 has
failed
So
spring.datasource.url=jdbc:sqlserver://JD1LSTWLSLMC101.dcsr.site:28001; . . .

sftp in apache camel tries kerberos authentication

My camel route picks up files from a sftp endpoint configured with username and password. Before the connection is made, i get a prompt for username and password from Kerberos.
I skip this by pressing enter, and authentication proceeds as expected, the files i picked up. Edit: After going through the versioning logs and the program logs i'm not entirely sure this is what happend, and i cannot reproduce it.
What i see it that my (one) screen-copy with the Kerberos prompt coincides with the one of several logs showing authentication failure because my username is wrong.
Why do i get the prompt? My endpoint looks like this:
sendEmptyMessageWhenIdle: sendEmptyMessageWhenIdle=true
useHostFile: knownHostsFile=known_hosts&strictHostKeyChecking=yes
workDirectory: /omg/myprogram/${env}/WorkDirectory
fromParams: streamDownload=true&maximumReconnectAttempts=2&throwExceptionOnConnectFailed=true&consumer.bridgeErrorHandler=true&disconnectOnBatchComplete=true
mypassword: password=blablabla
fromUri: sftp://myuser#accept.server.nu:22/TST?${mypassword}
from: ${fromUri}&${sendEmptyMessageWhenIdle}&${routeScheduler}&delete=true&${useHostFile}&localWorkDirectory=${workDirectory}&${fromParams}
Actually when i dont use a hosts file the same thing happens:
useUserKnownHostsFile=false
The camel version is 2.22.2.
What can i do to disable Kerberos in Apache camel?
Could you please set preferredAuthentications attribute on the SFTP component and see if makes a difference? This is an example preferredAuthentications=publickey,keyboard-interactive,password. Documentation for FTP component here. The docs say if you don't set this, the component will use default list of mechanisms in JSCH.
Edit:
I see a similar question here but in JSCH perspective.
It appears that the Kerberos prompt appears when authentication with the provided user-name and password fails.
That is, i have not been able to reproduce the problem i described above but i have found that the Kerberos prompt appeared when the username was incorrect.
Log:
2019-01-31 12:42:30,767 DEBUG [main] myprogram.StartUpVerneProphet - - - Running with Spring Boot v2.0.8.RELEASE, Spring v5.0.12.RELEASE
2019-01-31 12:42:30,769 INFO [main] myprogram.StartUpVerneProphet - - - No active profile set, falling back to default profiles: default
2019-01-31 12:42:36,003 WARN [main] org.apache.camel.component.file.remote.SftpOperations - - - JSCH -> Permanently added 'accept.server.nu' (DSA) to the list of known hosts.
2019-01-31 12:43:01,537 WARN [main] org.apache.camel.component.file.remote.SftpOperations - - - JSCH -> Permanently added 'accept.server.nu' (DSA) to the list of known hosts.
2019-01-31 12:44:13,940 WARN [main] org.apache.camel.component.file.remote.SftpOperations - - - JSCH -> Permanently added 'accept.server.nu' (DSA) to the list of known hosts.
2019-01-31 12:44:24,963 WARN [main] org.apache.camel.component.file.remote.SftpConsumer - - - Error auto creating directory: TST due Cannot connect to sftp://faulty-username#accept.server.nu:22. This exception is ignored. org.apache.camel.component.file.GenericFileOperationFailedException: Cannot connect to sftp://faulty-username#accept.server.nu:22
at org.apache.camel.component.file.remote.SftpOperations.connect(SftpOperations.java:144)
at org.apache.camel.component.file.remote.RemoteFileConsumer.connectIfNecessary(RemoteFileConsumer.java:197)
Screen-copy:
12:42:30,761 |-INFO in c.q.l.core.rolling.helper.TimeBasedArchiveRemover - first clean up after appender initialization
12:42:30,763 |-INFO in c.q.l.core.rolling.helper.TimeBasedArchiveRemover - Multiple periods, i.e. 32 periods, seem to have elapsed. This is expected a t application start.
2019-01-31 12:42:30 DEBUG myprogram.StartUp - - - Running with Spring Boot v2.0.8.RELEASE, Spr ing v5.0.12.RELEASE
2019-01-31 12:42:30 INFO myprogram.StartUp - - - No active profile set, falling back to defau lt profiles: default
2019-01-31 12:42:36 WARN org.apache.camel.component.file.remote.SftpOperations - - - JSCH -> Permanently added 'accept.server.nu' (DSA) to the list of known hosts.
Kerberos username [root]:
Kerberos password for root:
2019-01-31 12:43:01 WARN org.apache.camel.component.file.remote.SftpOperations - - - JSCH -> Permanently added 'accept.server.nu' (DSA) to the list of known hosts.
Kerberos username [root]:
Kerberos password for root:
Unfortunately, i made only one screencopy, so i am unsure if there was any occasion where the Kerberos prompt appeared after i used the correct user-name.
It hasn't appeared totday.

How to stop JSCH logs for camel sftp consumer

My camel SFTP consumer connects to server every 10 seconds and during every connect it prints huge log messages as mentioned here.
02-05-2016 20:48:34,441 INFO
(org.apache.camel.component.file.remote.SftpOperations) [Camel
(camel-99) thread #117 -
sftp://username:******#11.11.11.11:22/download] JSCH -> kex: client:
hmac-md5,hmac-sha1,hmac-sha1-96,hmac-md5-96
02-05-2016 20:48:34,441 INFO
(org.apache.camel.component.file.remote.SftpOperations) [Camel
(camel-99) thread #117 -
sftp://username:******#11.11.11.11:22/download] JSCH -> kex: client:
none
02-05-2016 20:48:34,441 INFO
(org.apache.camel.component.file.remote.SftpOperations) [Camel
(camel-99) thread #117 -
sftp://username:******#11.11.11.11:22/download] JSCH -> kex: client:
none
02-05-2016 20:48:34,441 INFO
(org.apache.camel.component.file.remote.SftpOperations) [Camel
(camel-99) thread #117 -
sftp://username:******#11.11.11.11:22/download] JSCH -> kex: client:
02-05-2016 20:48:34,441 INFO
(org.apache.camel.component.file.remote.SftpOperations) [Camel
(camel-99) thread #117 -
sftp://username:******#11.11.11.11:22/download] JSCH -> kex: client:
02-05-2016 20:48:34,441 INFO
(org.apache.camel.component.file.remote.SftpOperations) [Camel
(camel-99) thread #117 -
sftp://username:******#11.11.11.11:22/download] JSCH -> kex:
server->client aes128-ctr hmac-md5 none
02-05-2016 20:48:34,441 INFO
(org.apache.camel.component.file.remote.SftpOperations) [Camel
(camel-99) thread #117 -
sftp://username:******#11.11.11.11:22/download] JSCH -> kex:
client->server aes128-ctr hmac-md5 none
02-05-2016 20:48:34,443 INFO
(org.apache.camel.component.file.remote.SftpOperations) [Camel
(camel-99) thread #117 -
sftp://username:******#11.11.11.11:22/download] JSCH ->
SSH_MSG_KEXDH_INIT sent
02-05-2016 20:48:34,444 INFO
(org.apache.camel.component.file.remote.SftpOperations) [Camel
(camel-99) thread #117 -
sftp://username:******#11.11.11.11:22/download] JSCH -> expecting
SSH_MSG_KEXDH_REPLY
02-05-2016 20:48:34,762 INFO
(org.apache.camel.component.file.remote.SftpOperations) [Camel
(camel-99) thread #117 -
sftp://username:******#11.11.11.11:22/download] JSCH ->
ssh_rsa_verify: signature true
02-05-2016 20:48:34,762 WARN
(org.apache.camel.component.file.remote.SftpOperations) [Camel
(camel-99) thread #117 -
sftp://username:******#11.11.11.11:22/download] JSCH -> Permanently
added '11.11.11.11' (RSA) to the list of known hosts.
02-05-2016 20:48:34,763 INFO
(org.apache.camel.component.file.remote.SftpOperations) [Camel
(camel-99) thread #117 -
sftp://username:******#11.11.11.11:22/download] JSCH ->
SSH_MSG_NEWKEYS sent
02-05-2016 20:48:34,763 INFO
(org.apache.camel.component.file.remote.SftpOperations) [Camel
(camel-99) thread #117 -
sftp://username:******#11.11.11.11:22/download] JSCH ->
SSH_MSG_NEWKEYS received
02-05-2016 20:48:34,763 INFO
(org.apache.camel.component.file.remote.SftpOperations) [Camel
(camel-99) thread #117 -
sftp://username:******#11.11.11.11:22/download] JSCH ->
SSH_MSG_SERVICE_REQUEST sent
02-05-2016 20:48:35,079 INFO
(org.apache.camel.component.file.remote.SftpOperations) [Camel
(camel-99) thread #117 -
sftp://username:******#11.11.11.11:22/download] JSCH ->
SSH_MSG_SERVICE_ACCEPT received
02-05-2016 20:48:35,391 INFO
(org.apache.camel.component.file.remote.SftpOperations) [Camel
(camel-99) thread #117 -
sftp://username:******#11.11.11.11:22/download] JSCH ->
Authentications that can continue:
publickey,keyboard-interactive,password
02-05-2016 20:48:35,391 INFO
(org.apache.camel.component.file.remote.SftpOperations) [Camel
(camel-99) thread #117 -
sftp://username:******#11.11.11.11:22/download] JSCH -> Next
authentication method: publickey
02-05-2016 20:48:35,391 INFO
(org.apache.camel.component.file.remote.SftpOperations) [Camel
(camel-99) thread #117 -
sftp://username:******#11.11.11.11:22/download] JSCH ->
Authentications that can continue: password
02-05-2016 20:48:35,391 INFO
(org.apache.camel.component.file.remote.SftpOperations) [Camel
(camel-99) thread #117 -
sftp://username:******#11.11.11.11:22/download] JSCH -> Next
authentication method: password
02-05-2016 20:48:35,707 INFO
(org.apache.camel.component.file.remote.SftpOperations) [Camel
(camel-99) thread #117 -
sftp://username:******#11.11.11.11:22/download] JSCH -> Authentication
succeeded (password).
02-05-2016 20:48:36,662 INFO
(org.apache.camel.component.file.remote.SftpOperations) [Camel
(camel-99) thread #117 -
sftp://username:******#11.11.11.11:22/download] Connected to
sftp://username#11.11.11.11:22
02-05-2016 20:48:36,662 INFO
(org.apache.camel.component.file.remote.SftpConsumer) [Camel
(camel-99) thread #117 -
sftp://username:******#11.11.11.11:22/download] Connected and logged
in to: sftp://username#11.11.11.11:22
02-05-2016 20:48:38,539 INFO
(org.apache.camel.component.file.remote.SftpOperations) [Camel
(camel-99) thread #117 -
sftp://username:******#11.11.11.11:22/download] JSCH -> Disconnecting
from 11.11.11.11 port 22
How can I stop these messages from appearing in the log?
If you are using a logger like log4j, you can simply turn off the JSCH logging with the following line in your logger config:
log4j.logger.org.apache.camel.component.file.remote.SftpOperations = OFF
As an alternative to change the logger config, the current Camel sFTP documentation mentions few options you can set on the endpoint:
jschLoggingLevel
readLockLoggingLevel
runLoggingLevel
For example you can set:
sftp://user#myserver.mydomain.net:9921/path/to/folder?binary=true&jschLoggingLevel=WARN&readLockLoggingLevel=INFO&runLoggingLevel=WARN
You can tweak each category to fit your needs directly in the endpoint without changing the configuration of your logger.

zookeeper does not run?

I wanted to run a solr cloud with solr 4.3.0.
(I am using aws ubuntu-12.04-lts micro instances)
So I followed this toturial:
which basically says, start the zookeeper and connect the solr instances to it.
Here's how I start the zookeeper.
First I copied the config like described in the tutorial
sudo cp zookeeper-3.4.5/conf/zoo_sample.cfg zookeeper-3.4.5/conf/zoo.cfg
Then I started the zookeeper
ubuntu#ip-10-48-159-36:/opt$ sudo zookeeper-3.4.5/bin/zkServer.sh start
JMX enabled by default
Using config: /opt/zookeeper-3.4.5/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
Looks fine so far.
I checked the status:
ubuntu#ip-10-48-159-36:/opt$ sudo zookeeper-3.4.5/bin/zkServer.sh status
JMX enabled by default
Using config: /opt/zookeeper-3.4.5/bin/../conf/zoo.cfg
Error contacting service. It is probably not running.
Which seems a bit weird already.
If I try to connect with the client (remote as well as local), its seems to work
ubuntu#ip-10-234-223-69:/opt$ zookeeper-3.4.5/bin/zkCli.sh -server ec2-54-247-144-120.eu-west-1.compute.amazonaws.com:2181
Connecting to ec2-54-247-144-120.eu-west-1.compute.amazonaws.com:2181
2013-06-07 11:07:01,996 [myid:] - INFO [main:Environment#100] - Client environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
2013-06-07 11:07:02,000 [myid:] - INFO [main:Environment#100] - Client environment:host.name=ip-10-234-223-69.eu-west-1.compute.internal
2013-06-07 11:07:02,000 [myid:] - INFO [main:Environment#100] - Client environment:java.version=1.6.0_27
2013-06-07 11:07:02,002 [myid:] - INFO [main:Environment#100] - Client environment:java.vendor=Sun Microsystems Inc.
2013-06-07 11:07:02,003 [myid:] - INFO [main:Environment#100] - Client environment:java.home=/usr/lib/jvm/java-6-openjdk-amd64/jre
2013-06-07 11:07:02,003 [myid:] - INFO [main:Environment#100] - Client environment:java.class.path=/opt/zookeeper-3.4.5/bin/../build/classes:/opt/zookeeper-3.4.5/bin/../build/lib/*.jar:/opt/zookeeper-3.4.5/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/zookeeper-3.4.5/bin/../lib/slf4j-api-1.6.1.jar:/opt/zookeeper-3.4.5/bin/../lib/netty-3.2.2.Final.jar:/opt/zookeeper-3.4.5/bin/../lib/log4j-1.2.15.jar:/opt/zookeeper-3.4.5/bin/../lib/jline-0.9.94.jar:/opt/zookeeper-3.4.5/bin/../zookeeper-3.4.5.jar:/opt/zookeeper-3.4.5/bin/../src/java/lib/*.jar:/opt/zookeeper-3.4.5/bin/../conf:
2013-06-07 11:07:02,004 [myid:] - INFO [main:Environment#100] - Client environment:java.library.path=/usr/lib/jvm/java-6-openjdk-amd64/jre/lib/amd64/server:/usr/lib/jvm/java-6-openjdk-amd64/jre/lib/amd64:/usr/lib/jvm/java-6-openjdk-amd64/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
2013-06-07 11:07:02,008 [myid:] - INFO [main:Environment#100] - Client environment:java.io.tmpdir=/tmp
2013-06-07 11:07:02,009 [myid:] - INFO [main:Environment#100] - Client environment:java.compiler=<NA>
2013-06-07 11:07:02,018 [myid:] - INFO [main:Environment#100] - Client environment:os.name=Linux
2013-06-07 11:07:02,019 [myid:] - INFO [main:Environment#100] - Client environment:os.arch=amd64
2013-06-07 11:07:02,019 [myid:] - INFO [main:Environment#100] - Client environment:os.version=3.2.0-40-virtual
2013-06-07 11:07:02,020 [myid:] - INFO [main:Environment#100] - Client environment:user.name=ubuntu
2013-06-07 11:07:02,020 [myid:] - INFO [main:Environment#100] - Client environment:user.home=/home/ubuntu
2013-06-07 11:07:02,021 [myid:] - INFO [main:Environment#100] - Client environment:user.dir=/opt
2013-06-07 11:07:02,029 [myid:] - INFO [main:ZooKeeper#438] - Initiating client connection, connectString=ec2-54-247-144-120.eu-west-1.compute.amazonaws.com:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher#182d9c06
Welcome to ZooKeeper!
2013-06-07 11:07:02,074 [myid:] - INFO [main-SendThread(ip-10-48-159-36.eu-west-1.compute.internal:2181):ClientCnxn$SendThread#966] - Opening socket connection to server ip-10-48-159-36.eu-west-1.compute.internal/10.48.159.36:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
[zk: ec2-54-247-144-120.eu-west-1.compute.amazonaws.com:2181(CONNECTING) 0] 2013-06-07 11:07:32,100 [myid:] - INFO [main-SendThread(ip-10-48-159-36.eu-west-1.compute.internal:2181):ClientCnxn$SendThread#1083] - Client session timed out, have not heard from server in 30038ms for sessionid 0x0, closing socket connection and attempting reconnect
2013-06-07 11:07:33,204 [myid:] - INFO [main-SendThread(ip-10-48-159-36.eu-west-1.compute.internal:2181):ClientCnxn$SendThread#966] - Opening socket connection to server ip-10-48-159-36.eu-west-1.compute.internal/10.48.159.36:2181. Will not attempt to authenticate using SASL (unknown error)
Now I tried to connect a solr instance to it. In the web interface of tomcat7 it only tells me "503 - Server is shutting down", so I checked the solr logs
2013-06-07 11:16:36,065 [pool-2-thread-1] INFO org.apache.solr.servlet.SolrDispatchFilter . SolrDispatchFilter.init()
2013-06-07 11:16:36,100 [pool-2-thread-1] INFO org.apache.solr.core.SolrResourceLoader . Using JNDI solr.home: /opt/solr-4.3.0/example/solr
2013-06-07 11:16:36,132 [pool-2-thread-1] INFO org.apache.solr.core.CoreContainer . looking for solr config file: /opt/solr-4.3.0/example/solr/solr.xml
2013-06-07 11:16:36,138 [pool-2-thread-1] INFO org.apache.solr.core.CoreContainer . New CoreContainer 1285984216
2013-06-07 11:16:36,146 [pool-2-thread-1] INFO org.apache.solr.core.CoreContainer . Loading CoreContainer using Solr Home: '/opt/solr-4.3.0/example/solr/'
2013-06-07 11:16:36,152 [pool-2-thread-1] INFO org.apache.solr.core.SolrResourceLoader . new SolrResourceLoader for directory: '/opt/solr-4.3.0/example/solr/'
2013-06-07 11:16:36,567 [pool-2-thread-1] INFO org.apache.solr.handler.component.HttpShardHandlerFactory . Setting socketTimeout to: 0
2013-06-07 11:16:36,568 [pool-2-thread-1] INFO org.apache.solr.handler.component.HttpShardHandlerFactory . Setting urlScheme to: http://
2013-06-07 11:16:36,568 [pool-2-thread-1] INFO org.apache.solr.handler.component.HttpShardHandlerFactory . Setting connTimeout to: 0
2013-06-07 11:16:36,568 [pool-2-thread-1] INFO org.apache.solr.handler.component.HttpShardHandlerFactory . Setting maxConnectionsPerHost to: 20
2013-06-07 11:16:36,568 [pool-2-thread-1] INFO org.apache.solr.handler.component.HttpShardHandlerFactory . Setting corePoolSize to: 0
2013-06-07 11:16:36,568 [pool-2-thread-1] INFO org.apache.solr.handler.component.HttpShardHandlerFactory . Setting maximumPoolSize to: 2147483647
2013-06-07 11:16:36,568 [pool-2-thread-1] INFO org.apache.solr.handler.component.HttpShardHandlerFactory . Setting maxThreadIdleTime to: 5
2013-06-07 11:16:36,569 [pool-2-thread-1] INFO org.apache.solr.handler.component.HttpShardHandlerFactory . Setting sizeOfQueue to: -1
2013-06-07 11:16:36,569 [pool-2-thread-1] INFO org.apache.solr.handler.component.HttpShardHandlerFactory . Setting fairnessPolicy to: false
2013-06-07 11:16:36,578 [pool-2-thread-1] INFO org.apache.solr.client.solrj.impl.HttpClientUtil . Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
2013-06-07 11:16:36,879 [pool-2-thread-1] INFO org.apache.solr.core.CoreContainer . Registering Log Listener
2013-06-07 11:16:36,881 [pool-2-thread-1] INFO org.apache.solr.core.CoreContainer . Zookeeper client=ec2-54-247-144-120.eu-west-1.compute.amazonaws.com:2181
2013-06-07 11:16:36,888 [pool-2-thread-1] INFO org.apache.solr.client.solrj.impl.HttpClientUtil . Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=0&connTimeout=0
2013-06-07 11:16:37,040 [pool-2-thread-1] INFO org.apache.solr.common.cloud.ConnectionManager . Waiting for client to connect to ZooKeeper
2013-06-07 11:16:52,046 [pool-2-thread-1] ERROR org.apache.solr.servlet.SolrDispatchFilter . Could not start Solr. Check solr/home property and the logs
2013-06-07 11:16:52,103 [pool-2-thread-1] ERROR org.apache.solr.core.SolrCore . null:java.lang.RuntimeException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper ec2-54-247-144-120.eu-west-1.compute.amazonaws.com:2181 within 15000 ms
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:130)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:88)
at org.apache.solr.cloud.ZkController.<init>(ZkController.java:170)
at org.apache.solr.core.CoreContainer.initZooKeeper(CoreContainer.java:242)
at org.apache.solr.core.CoreContainer.load(CoreContainer.java:495)
at org.apache.solr.core.CoreContainer.load(CoreContainer.java:358)
at org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:326)
at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:124)
at org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:277)
at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:258)
at org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:382)
at org.apache.catalina.core.ApplicationFilterConfig.<init>(ApplicationFilterConfig.java:103)
at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4638)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5294)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:895)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:871)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:615)
at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:649)
at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1581)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper ec2-54-247-144-120.eu-west-1.compute.amazonaws.com:2181 within 15000 ms
at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:173)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:127)
... 25 more
2013-06-07 11:16:52,104 [pool-2-thread-1] INFO org.apache.solr.servlet.SolrDispatchFilter . SolrDispatchFilter.init() done
What does it tell me?
On the same instance I just connected with the client successfully... :(
So where is the problem?
[Edit:]
Instead of using amazons ec**.amazon.* address I used the network addresses 10.X.X.X for telling solr where the zookeeper is.
It seems to work.
You have your answer - Your ZooKeeper in inaccessible!
Check your firewall configuration.
You can also check it with
zkCli.sh -server localhost:2181
There must have been some sort of connectivity problem.
I see you have it resolved now.
Next time you run into a situation like this, you should log onto the box that is having problems connecting and use telnet to see if you can connect.
eg: from your solr box:
telnet ec2-54-247-144-120.eu-west-1.compute.amazonaws.com 2181
and then try from the zk box too. It should start to illuminate where your issues are.
That eliminates any application layer issues and will tell you quite reliably wether or not you can connect. It you can't connect, then it's almost always some sort of security issue - either a firewall running somewhere (try - $service iptables stop) or it will be an issue with security group configuration in amazon.
The last potential problem is network availability. Despite what people think, the network is NOT reliable and should never be considered so. Anyone working in SOA/distributed systems will know this well :)
http://aphyr.com/posts/288-the-network-is-reliable
"A team from the University of Toronto and Microsoft Research studied the behavior of network failures in several of Microsoft’s datacenters. They found an average failure rate of 5.2 devices per day and 40.8 links per day with a median time to repair of approximately five minutes (and up to one week). "
While setting up SolrCloud and ZooKeeper I also ran into the "Error contacting service. It is probably not running." issue. The reason was a typo in a file name that ZooKeeper needs. The correct file name is "myid". I wrote "myip" by mistake. After the renaming of the file and restarting ZooKeeper (./zkServer.sh restart), my issue was resolved.
try to stop your solr instance solr.shutdown() so that you can create new CloudSolrServer instance for each thread

Resources