Weblogic : Even After supplying Custom HostName Verifier---- Still Same SSL Issue - weblogic11g
Experts !
I had been struggling to make my Webservice Client (using https URL) work within Weblogic Server 10.3.2, which had been causing an issue as below:
Feb 3, 2012 10:50:28 AM EST Warning Security BEA-090504 Certificate chain received from apcple.XXX.com - 113.128.90.16 failed hostname verification check. Certificate contained apcple.XXX.com but check expected apcple.XXX.com
If i disable host name verification check using -Dweblogic.security.SSL.ignoreHostnameVerification=true , Code is WORKING fine. No Issue. But that is NOT what is want.
a) I enabled weblogic SSL debugs.
b) I introduced my Custom HostNameVerifier and supplied it via console , I see in the logs that My custom verifier is getting picked up and does Host Name comparison against URL hostname Vs Certiicate CN name and it suceeds and code returns true.
c) If you see SSL debug statements, Connection with server got established and First time it tries to validate Certificate chain 0 ,1, 2 and LOADS my CUSTOM HostName Verifier.
d) After few lines passed , It again tries to validate Certificates in same Series and SECOND time it DID NOT load My Custome HostName Verifier AND FAILS with the Standard BEA Security Error as pasted above.
Would you please help me to figure out what is missing ?
Any help in this regard would be highly appreciated.
SEE SNIPPET OF SSL DEBUG STATEMENTS
Feb 3, 2012 10:50:23 AM EST Info WebLogicServer BEA-000307 Exportable key maximum lifespan set to 500 uses.
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 Filtering JSSE SSLSocket
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 SSLIOContextTable.addContext(ctx): 30958379
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 SSLSocket will be Muxing
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 write SSL_20_RECORD
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 isMuxerActivated: false
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 746666 SSL3/TLS MAC
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 746666 received HANDSHAKE
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 HANDSHAKEMESSAGE: ServerHello
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 HANDSHAKEMESSAGE: Certificate
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 Validating certificate 0 in the chain: Serial number: 1208925819615937499602513
Issuer:O=Cybertrust Inc, CN=Cybertrust Public SureServer SV CA
Subject:C=US, ST=Texas, L=Irving, O=XXX LLC, OU=ns, EMAIL=aes.eng#XXX.com, CN=apcple.XXX.com
Not Valid Before:Fri Jun 17 10:48:17 EDT 2011
Not Valid After:Sun Jun 17 10:48:17 EDT 2012
Signature Algorithm:SHA1withRSA
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 Validating certificate 1 in the chain: Serial number: 120010508
Issuer:C=IE, O=Baltimore, OU=CyberTrust, CN=Baltimore CyberTrust Root
Subject:O=Cybertrust Inc, CN=Cybertrust Public SureServer SV CA
Not Valid Before:Wed Sep 08 13:35:16 EDT 2010
Not Valid After:Tue Sep 08 13:34:08 EDT 2020
Signature Algorithm:SHA1withRSA
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 Validating certificate 2 in the chain: Serial number: 33554617
Issuer:C=IE, O=Baltimore, OU=CyberTrust, CN=Baltimore CyberTrust Root
Subject:C=IE, O=Baltimore, OU=CyberTrust, CN=Baltimore CyberTrust Root
Not Valid Before:Fri May 12 14:46:00 EDT 2000
Not Valid After:Mon May 12 19:59:00 EDT 2025
Signature Algorithm:SHA1withRSA
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 validationCallback: validateErr = 0
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 cert[0] = Serial number: 1208925819615937499602513
Issuer:O=Cybertrust Inc, CN=Cybertrust Public SureServer SV CA
Subject:C=US, ST=Texas, L=Irving, O=XXX LLC, OU=ns, EMAIL=aes.eng#XXX.com, CN=apcple.XXX.com
Not Valid Before:Fri Jun 17 10:48:17 EDT 2011
Not Valid After:Sun Jun 17 10:48:17 EDT 2012
Signature Algorithm:SHA1withRSA
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 cert[1] = Serial number: 120010508
Issuer:C=IE, O=Baltimore, OU=CyberTrust, CN=Baltimore CyberTrust Root
Subject:O=Cybertrust Inc, CN=Cybertrust Public SureServer SV CA
Not Valid Before:Wed Sep 08 13:35:16 EDT 2010
Not Valid After:Tue Sep 08 13:34:08 EDT 2020
Signature Algorithm:SHA1withRSA
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 cert[2] = Serial number: 33554617
Issuer:C=IE, O=Baltimore, OU=CyberTrust, CN=Baltimore CyberTrust Root
Subject:C=IE, O=Baltimore, OU=CyberTrust, CN=Baltimore CyberTrust Root
Not Valid Before:Fri May 12 14:46:00 EDT 2000
Not Valid After:Mon May 12 19:59:00 EDT 2025
Signature Algorithm:SHA1withRSA
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 weblogic user specified trustmanager validation status 0
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 SSLTrustValidator returns: 0
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 Trust status (0): NONE
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 Performing hostname validation checks: apcple.XXX.com
############ BELOW 6 LINES INDICATES MY CUSTOM HOST NAME VERIFIER GOT PICKED UP ########
Custom HostName Verifier Called =com.XXX.sys.WeblogicHostNameVerifier
Rcvd. Host Name=apcple.XXX.com SSL Session=javax.net.ssl.impl.SSLSessionImpl#1760238
Parsing COMMON Name from Certificates
getPeerLeafCert()....Start
Parsed CN = apcple.XXX.com
HOST NAME AND COMMON NAME MATCH FOUND
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 HANDSHAKEMESSAGE: ServerHelloDone
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 Ignoring not supported JCE Mac: SunJCE version 1.6 for algorithm HmacMD5
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 Will use default Mac for algorithm HmacMD5
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 Ignoring not supported JCE Mac: SunJCE version 1.6 for algorithm HmacSHA1
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 Will use default Mac for algorithm HmacSHA1
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 Will use default Mac for algorithm MD5
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 Using JCE Cipher: SunJCE version 1.6 for algorithm RC4
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 Ignoring not supported JCE Mac: SunJCE version 1.6 for algorithm HmacMD5
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 Will use default Mac for algorithm HmacMD5
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 Ignoring not supported JCE Mac: SunJCE version 1.6 for algorithm HmacSHA1
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 Will use default Mac for algorithm HmacSHA1
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 Using JCE Cipher: SunJCE version 1.6 for algorithm RSA
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 write HANDSHAKE, offset = 0, length = 262
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 write CHANGE_CIPHER_SPEC, offset = 0, length = 1
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 Using JCE Cipher: SunJCE version 1.6 for algorithm RC4
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 Ignoring not supported JCE Mac: SunJCE version 1.6 for algorithm HMACMD5
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 Will use default Mac for algorithm HMACMD5
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 Ignoring not supported JCE Mac: SunJCE version 1.6 for algorithm HmacMD5
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 Will use default Mac for algorithm HmacMD5
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 Ignoring not supported JCE Mac: SunJCE version 1.6 for algorithm HmacSHA1
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 Will use default Mac for algorithm HmacSHA1
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 write HANDSHAKE, offset = 0, length = 16
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 isMuxerActivated: false
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 746666 SSL3/TLS MAC
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 746666 received CHANGE_CIPHER_SPEC
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 Using JCE Cipher: SunJCE version 1.6 for algorithm RC4
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 Ignoring not supported JCE Mac: SunJCE version 1.6 for algorithm HMACMD5
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 Will use default Mac for algorithm HMACMD5
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 isMuxerActivated: false
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 746666 SSL3/TLS MAC
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 746666 received HANDSHAKE
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 HANDSHAKEMESSAGE: Finished
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 Ignoring not supported JCE Mac: SunJCE version 1.6 for algorithm HmacMD5
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 Will use default Mac for algorithm HmacMD5
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 Ignoring not supported JCE Mac: SunJCE version 1.6 for algorithm HmacSHA1
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 Will use default Mac for algorithm HmacSHA1
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 write APPLICATION_DATA, offset = 0, length = 193
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 31771821 read(offset=0, length=8192)
Feb 3, 2012 10:50:23 AM EST Debug SecuritySSL BEA-000000 isMuxerActivated: false
Feb 3, 2012 10:50:24 AM EST Debug SecuritySSL BEA-000000 746666 SSL3/TLS MAC
Feb 3, 2012 10:50:24 AM EST Debug SecuritySSL BEA-000000 746666 received APPLICATION_DATA: databufferLen 0, contentLength 143
Feb 3, 2012 10:50:24 AM EST Debug SecuritySSL BEA-000000 31771821 read databufferLen 143
Feb 3, 2012 10:50:24 AM EST Debug SecuritySSL BEA-000000 31771821 read A returns 143
Feb 3, 2012 10:50:24 AM EST Debug SecuritySSL BEA-000000 31771821 read(offset=143, length=8049)
Feb 3, 2012 10:50:24 AM EST Debug SecuritySSL BEA-000000 isMuxerActivated: false
Feb 3, 2012 10:50:24 AM EST Debug SecuritySSL BEA-000000 746666 SSL3/TLS MAC
Feb 3, 2012 10:50:24 AM EST Debug SecuritySSL BEA-000000 746666 received APPLICATION_DATA: databufferLen 0, contentLength 3819
Feb 3, 2012 10:50:24 AM EST Debug SecuritySSL BEA-000000 31771821 read databufferLen 3819
Feb 3, 2012 10:50:24 AM EST Debug SecuritySSL BEA-000000 31771821 read A returns 3819
Feb 3, 2012 10:50:24 AM EST Debug SecuritySSL BEA-000000 31771821 read(offset=3962, length=4230)
Feb 3, 2012 10:50:24 AM EST Debug SecuritySSL BEA-000000 isMuxerActivated: false
Feb 3, 2012 10:50:24 AM EST Debug SecuritySSL BEA-000000 746666 SSL3/TLS MAC
Feb 3, 2012 10:50:24 AM EST Debug SecuritySSL BEA-000000 746666 received APPLICATION_DATA: databufferLen 0, contentLength 8
Feb 3, 2012 10:50:24 AM EST Debug SecuritySSL BEA-000000 31771821 read databufferLen 8
Feb 3, 2012 10:50:24 AM EST Debug SecuritySSL BEA-000000 31771821 read A returns 8
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 write APPLICATION_DATA, offset = 0, length = 193
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 31771821 read(offset=3970, length=4222)
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 isMuxerActivated: false
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 746666 SSL3/TLS MAC
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 746666 received APPLICATION_DATA: databufferLen 0, contentLength 143
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 31771821 read databufferLen 143
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 31771821 read A returns 143
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 31771821 read(offset=4113, length=4079)
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 isMuxerActivated: false
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 746666 SSL3/TLS MAC
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 746666 received APPLICATION_DATA: databufferLen 0, contentLength 3819
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 31771821 read databufferLen 3819
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 31771821 read A returns 3819
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 31771821 read(offset=7932, length=260)
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 isMuxerActivated: false
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 746666 SSL3/TLS MAC
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 746666 received APPLICATION_DATA: databufferLen 0, contentLength 8
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 31771821 read databufferLen 8
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 31771821 read A returns 8
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 SSLSetup: loading trusted CA certificates
Feb 3, 2012 10:50:25 AM EST Notice Security BEA-090898 Ignoring the trusted CA certificate "CN=T-TeleSec GlobalRoot Class 3,OU=T-Systems Trust Center,O=T-Systems Enterprise Services GmbH,C=DE". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.
Feb 3, 2012 10:50:25 AM EST Notice Security BEA-090898 Ignoring the trusted CA certificate "CN=T-TeleSec GlobalRoot Class 2,OU=T-Systems Trust Center,O=T-Systems Enterprise Services GmbH,C=DE". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 clientInfo has HostnameVerifier
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 Filtering JSSE SSLSocket
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 SSLIOContextTable.addContext(ctx): 25583909
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 SSLSocket will be Muxing
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 write SSL_20_RECORD
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 isMuxerActivated: false
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 28097422 SSL3/TLS MAC
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 28097422 received HANDSHAKE
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 HANDSHAKEMESSAGE: ServerHello
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 HANDSHAKEMESSAGE: Certificate
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 Validating certificate 0 in the chain: Serial number: 1208925819615937499602513
Issuer:O=Cybertrust Inc, CN=Cybertrust Public SureServer SV CA
Subject:C=US, ST=Texas, L=Irving, O=XXX LLC, OU=ns, EMAIL=aes.eng#XXX.com, CN=apcple.XXX.com
Not Valid Before:Fri Jun 17 10:48:17 EDT 2011
Not Valid After:Sun Jun 17 10:48:17 EDT 2012
Signature Algorithm:SHA1withRSA
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 Validating certificate 1 in the chain: Serial number: 120010508
Issuer:C=IE, O=Baltimore, OU=CyberTrust, CN=Baltimore CyberTrust Root
Subject:O=Cybertrust Inc, CN=Cybertrust Public SureServer SV CA
Not Valid Before:Wed Sep 08 13:35:16 EDT 2010
Not Valid After:Tue Sep 08 13:34:08 EDT 2020
Signature Algorithm:SHA1withRSA
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 Validating certificate 2 in the chain: Serial number: 33554617
Issuer:C=IE, O=Baltimore, OU=CyberTrust, CN=Baltimore CyberTrust Root
Subject:C=IE, O=Baltimore, OU=CyberTrust, CN=Baltimore CyberTrust Root
Not Valid Before:Fri May 12 14:46:00 EDT 2000
Not Valid After:Mon May 12 19:59:00 EDT 2025
Signature Algorithm:SHA1withRSA
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 validationCallback: validateErr = 0
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 cert[0] = Serial number: 1208925819615937499602513
Issuer:O=Cybertrust Inc, CN=Cybertrust Public SureServer SV CA
Subject:C=US, ST=Texas, L=Irving, O=XXX LLC, OU=ns, EMAIL=aes.eng#XXX.com, CN=apcple.XXX.com
Not Valid Before:Fri Jun 17 10:48:17 EDT 2011
Not Valid After:Sun Jun 17 10:48:17 EDT 2012
Signature Algorithm:SHA1withRSA
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 cert[1] = Serial number: 120010508
Issuer:C=IE, O=Baltimore, OU=CyberTrust, CN=Baltimore CyberTrust Root
Subject:O=Cybertrust Inc, CN=Cybertrust Public SureServer SV CA
Not Valid Before:Wed Sep 08 13:35:16 EDT 2010
Not Valid After:Tue Sep 08 13:34:08 EDT 2020
Signature Algorithm:SHA1withRSA
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 cert[2] = Serial number: 33554617
Issuer:C=IE, O=Baltimore, OU=CyberTrust, CN=Baltimore CyberTrust Root
Subject:C=IE, O=Baltimore, OU=CyberTrust, CN=Baltimore CyberTrust Root
Not Valid Before:Fri May 12 14:46:00 EDT 2000
Not Valid After:Mon May 12 19:59:00 EDT 2025
Signature Algorithm:SHA1withRSA
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 weblogic user specified trustmanager validation status 0
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 SSLTrustValidator returns: 0
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 Trust status (0): NONE
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 Performing hostname validation checks: apcple.XXX.com
Feb 3, 2012 10:50:25 AM EST Warning Security BEA-090504 Certificate chain received from apcple.XXX.com - 113.128.90.16 failed hostname verification check. Certificate contained apcple.XXX.com but check expected apcple.XXX.com
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 Hostname Verification failed for certificate with CommonName 'apcple.XXX.com' against hostname: apcple.XXX.com
Feb 3, 2012 10:50:25 AM EST Debug SecuritySSL BEA-000000 NEW ALERT with Severity: FATAL, Type: 42
java.lang.Exception: New alert stack
at com.certicom.tls.record.alert.Alert.init(Unknown Source)
at com.certicom.tls.record.handshake.HandshakeHandler.fireAlert(Unknown Source)
at com.certicom.tls.record.handshake.ClientStateReceivedServerHello.handle (Unknown Source)
at com.certicom.tls.record.handshake.HandshakeHandler.handleHandshakeMessage(Unknown Source)
at com.certicom.tls.record.handshake.HandshakeHandler.handleHandshakeMessages(Unknown Source)
at com.certicom.tls.record.MessageInterpreter.interpretContent(Unknown Source)
at com.certicom.tls.record.MessageInterpreter.decryptMessage(Unknown Source)
at com.certicom.tls.record.ReadHandler.processRecord(Unknown Source)
at com.certicom.tls.record.ReadHandler.readRecord(Unknown Source)
at com.certicom.tls.record.ReadHandler.readUntilHandshakeComplete(Unknown Source)
at com.certicom.tls.interfaceimpl.TLSConnectionImpl.completeHandshake(Unknown Source)
at com.certicom.tls.record.WriteHandler.write(Unknown Source)
at com.certicom.io.OutputSSLIOStreamWrapper.write(Unknown Source)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at java.io.FilterOutputStream.flush(FilterOutputStream.java:123)
at weblogic.net.http.HttpURLConnection.writeRequests(HttpURLConnection.java:158)
at weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:363)
at weblogic.net.http.SOAPHttpsURLConnection.getInputStream (SOAPHttpsURLConnection.java:37)
at weblogic.net.http.HttpURLConnection.getResponseCode(HttpURLConnection.java:952)
at com.sun.xml.ws.transport.http.client.HttpClientTransport.readResponseCodeAndMessage (HttpClientTransport.java:213)
at com.sun.xml.ws.transport.http.client.HttpTransportPipe.process (HttpTransportPipe.java:172)
at com.sun.xml.ws.transport.http.client.HttpTransportPipe.processRequest (HttpTransportPipe.java:101)
at com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:604)
at com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:563)
at com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:548)
at com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:445)
at com.sun.xml.ws.client.Stub.process(Stub.java:248)
at com.sun.xml.ws.client.sei.SEIStub.doProcess(SEIStub.java:135)
at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:109)
at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:89)
at com.sun.xml.ws.client.sei.SEIStub.invoke(SEIStub.java:118)
at $Proxy189.updateMilestone(Unknown Source)
at com.XXX.sys.apc.APCProcessor.updateMilestoneToApc(APCProcessor.java:95)
at com.XXX.sys.apc.APCProcessor.sendAsrFeed(APCProcessor.java:50)
at com.XXX.sys.apc.APCLookupBean.onMessage(APCLookupBean.java:94)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection (AopUtils.java:310)
at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoin point(ReflectiveMethodInvocation.java:182)
at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed (ReflectiveMethodInvocation.java:149)
at com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke (ExposeInvocationInterceptor.java:89)
at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed (ReflectiveMethodInvocation.java:171)
at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doPro ceed(DelegatingIntroductionInterceptor.java:131)
at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invok e(DelegatingIntroductionInterceptor.java:119)
at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed (ReflectiveMethodInvocation.java:171)
at com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke (JdkDynamicAopProxy.java:204)
at $Proxy148.onMessage(Unknown Source)
at weblogic.ejb.container.internal.MDListener.execute(MDListener.java:466)
at weblogic.ejb.container.internal.MDListener.transactionalOnMessage (MDListener.java:371)
at weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:327)
at weblogic.jms.client.JMSSession.onMessage(JMSSession.java:4585)
at weblogic.jms.client.JMSSession.execute(JMSSession.java:4271)
at weblogic.jms.client.JMSSession.executeMessage(JMSSession.java:3747)
at weblogic.jms.client.JMSSession.access$000(JMSSession.java:114)
at weblogic.jms.client.JMSSession$UseForRunnable.run(JMSSession.java:5096)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)'
Related
SQL Server 2022 in LXC crashes and won't get back up
I have big scary problem with SQL Server 2022 in LXC containers on PVE. SQL Server 2019 is working fine in same environment but SQL Server 2022 crashes after few months of working good. After crash it can't be restarted only solution I found is to reinstall the container. Feb 19 04:30:58 sqll sqlservr[4857]: This program has encountered a fatal error and cannot continue running at Sun Feb 19 04:30:58 2023 Feb 19 04:30:58 sqll sqlservr[4857]: The following diagnostic information is available: Feb 19 04:30:58 sqll sqlservr[4857]: Reason: 0x00000001 Feb 19 04:30:58 sqll sqlservr[4857]: Signal: SIGABRT - Aborted (6) Feb 19 04:30:58 sqll sqlservr[4857]: Stack: Feb 19 04:30:58 sqll sqlservr[4857]: IP Function Feb 19 04:30:58 sqll sqlservr[4857]: ---------------- -------------------------------------- Feb 19 04:30:58 sqll sqlservr[4857]: 0000562a9b7549d2 <unknown> Feb 19 04:30:58 sqll sqlservr[4857]: 0000562a9b7543f0 <unknown> Feb 19 04:30:58 sqll sqlservr[4857]: 0000562a9b753a8f <unknown> Feb 19 04:30:58 sqll sqlservr[4857]: 00007f093a98f090 killpg+0x40 Feb 19 04:30:58 sqll sqlservr[4857]: 00007f093a98f00b gsignal+0xcb Feb 19 04:30:58 sqll sqlservr[4857]: 00007f093a96e859 abort+0x12b Feb 19 04:30:58 sqll sqlservr[4857]: 0000562a9b70ec26 <unknown> Feb 19 04:30:58 sqll sqlservr[4857]: 0000562a9b78576a <unknown> Feb 19 04:30:58 sqll sqlservr[4857]: Process: 4857 - sqlservr Feb 19 04:30:58 sqll sqlservr[4857]: Thread: 4862 (application thread 0x8) Feb 19 04:30:58 sqll sqlservr[4857]: Instance Id: 1f101619-527d-4094-ac00-c2b857339367 Feb 19 04:30:58 sqll sqlservr[4857]: Crash Id: 8b1a4608-d26c-421a-beff-e83d1f30b91f Feb 19 04:30:58 sqll sqlservr[4857]: Build stamp: 7381104a7baabc096c65c6cf9b3c3c2c36e1f155ac27f817e87fab585602cb5f Feb 19 04:30:58 sqll sqlservr[4857]: Distribution: Ubuntu 20.04.5 LTS Feb 19 04:30:58 sqll sqlservr[4857]: Processors: 2 Feb 19 04:30:58 sqll sqlservr[4857]: Total Memory: 67350208512 bytes Feb 19 04:30:58 sqll sqlservr[4857]: Timestamp: Sun Feb 19 04:30:58 2023 Feb 19 04:30:58 sqll sqlservr[4857]: Last errno: 2 Feb 19 04:30:58 sqll sqlservr[4857]: Last errno text: No such file or directory Feb 19 04:30:58 sqll sqlservr[4855]: Capturing a dump of 4857 Feb 19 04:30:58 sqll sqlservr[4855]: FAILED to capture a dump. Details in paldumper log. Feb 19 04:30:58 sqll sqlservr[4855]: Executing: /opt/mssql/bin/handle-crash.sh with parameters Feb 19 04:30:58 sqll sqlservr[4855]: handle-crash.sh Feb 19 04:30:58 sqll sqlservr[4855]: /opt/mssql/bin/sqlservr Feb 19 04:30:58 sqll sqlservr[4855]: 4857 Feb 19 04:30:58 sqll sqlservr[4855]: /opt/mssql/bin Feb 19 04:30:58 sqll sqlservr[4855]: /var/opt/mssql/log/ Feb 19 04:30:58 sqll sqlservr[4855]: Feb 19 04:30:58 sqll sqlservr[4855]: 1f101619-527d-4094-ac00-c2b857339367 Feb 19 04:30:58 sqll sqlservr[4855]: 8b1a4608-d26c-421a-beff-e83d1f30b91f Feb 19 04:30:58 sqll sqlservr[4855]: Feb 19 04:30:58 sqll sqlservr[4855]: Feb 19 04:30:58 sqll sqlservr[4855]: Ubuntu 20.04.5 LTS This probably needs attention from somebody in msft. Related reports: https://github.com/MicrosoftDocs/sql-docs/issues/8525 https://forum.proxmox.com/threads/sqlserver-2022-lxc.120898/
RPC_HEADER error while phoenix from Zeppelin trying to connect
Installed zeppelin 0.8.0 and added phoenix interpreter ,
when I start querying for the data I am getting this below error , if I try from zeppelin 0.7.3 (adding phoenix interpreter) everything works fine
I am facing this error only when I tried from zeppelin 0.8.0
Query:
%pheonix
select * from "tablestest"
Error:
org.apache.phoenix.exception.PhoenixIOException: Failed after
attempts=35, exceptions:
Wed Oct 18 14:22:59 UTC 2017, RpcRetryingCaller{globalStartTime=1508336579165, pause=100, retries=35}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.NoSuchFieldError: RPC_HEADER
Wed Oct 18 14:22:59 UTC 2017, RpcRetryingCaller{globalStartTime=1508336579165, pause=100, retries=35}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: master/10.20.3.207:16000
Wed Oct 18 14:22:59 UTC 2017, RpcRetryingCaller{globalStartTime=1508336579165, pause=100, retries=35}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: master/10.20.3.207:16000
Wed Oct 18 14:23:00 UTC 2017, RpcRetryingCaller{globalStartTime=1508336579165, pause=100, retries=35}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: master/10.20.3.207:16000
Wed Oct 18 14:23:01 UTC 2017, RpcRetryingCaller{globalStartTime=1508336579165, pause=100, retries=35}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.NoSuchFieldError: RPC_HEADER
Wed Oct 18 14:23:03 UTC 2017, RpcRetryingCaller{globalStartTime=1508336579165, pause=100, retries=35}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.NoSuchFieldError: RPC_HEADER
Wed Oct 18 14:23:07 UTC 2017, RpcRetryingCaller{globalStartTime=1508336579165, pause=100, retries=35}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.NoSuchFieldError: RPC_HEADER
Solr 3,6 Replication Logs out of sync
I have replication on Solr 3.6 and I want to look at the logs because I have some issue with the timing. But looking there make me even more confused.
Solr Master Log:
Oct 8, 2015 6:46:36 AM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/replication params={command=filelist&wt=javabin&indexversion=1346489020771} status=0 QTime=5
Oct 8, 2015 6:46:36 AM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/replication params={command=filecontent&checksum=true&indexversion=1346489020771&wt=filestream&file=_6tyd.tii} status=0 QTime=0
Solr Slave 1:
INFO: Master's version: 1346489020771, generation: 288536
Oct 8, 2015 6:46:10 AM org.apache.solr.handler.SnapPuller fetchLatestIndex
INFO: Slave's version: 1346489020763, generation: 288534
...
...
Oct 8, 2015 6:46:11 AM org.apache.solr.handler.SnapPuller downloadIndexFiles
INFO: Skipping download for /solr/solr-home/cores_data/monitoring/index.20140415073723/_6tya.tvd
Oct 8, 2015 6:46:11 AM org.apache.solr.handler.SnapPuller fetchLatestIndex
INFO: Total time taken for download : 0 secs
As you can see the replication on the slave finish before the one on the master has even started.
Is this a problem with the index version numbers or a problem with the watch on the servers?
solr ReplicationHandler - SnapPull failed to download files
we are continuously getting this exception during replication from master to slave. our index size is 9.7 G and we are trying to replicate a slave from scratch. 30 Oct 2013 18:22:16,996 [explicit-fetchindex-cmd] ERROR ReplicationHandler - SnapPull failed :org.apache.solr.common.SolrException: Unable to download _41c_Lucene41_0.doc completely. Downloaded 0!=107464871 at org.apache.solr.handler.SnapPuller$DirectoryFileFetcher.cleanup(SnapPuller.java:1266) at org.apache.solr.handler.SnapPuller$DirectoryFileFetcher.fetchFile(SnapPuller.java:1146) at org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:741) at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:405) at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:319) at org.apache.solr.handler.ReplicationHandler$1.run(ReplicationHandler.java:220) I read in some thread that there was a related bug in solr 4.1, but we are using solr 4.3 and tried with 4.5.1 also. It seams that DirectoryFileFetcher can not download a file sometimes , the files is downloaded to the salve in size zero. this is the master setup: <requestHandler name="/replication" class="solr.ReplicationHandler" > <lst name="master"> <str name="replicateAfter">commit</str> <str name="replicateAfter">startup</str> <str name="confFiles">stopwords.txt,spellings.txt,synonyms.txt,protwords.txt,elevate.xml,currency.xml</str> <str name="commitReserveDuration">00:00:50</str> </lst> </requestHandler> and the slave setup: <requestHandler name="/replication" class="solr.ReplicationHandler" > <lst name="master"> <str name="replicateAfter">commit</str> <str name="replicateAfter">startup</str> <str name="confFiles">stopwords.txt,spellings.txt,synonyms.txt,protwords.txt,elevate.xml,currency.xml</str> <str name="commitReserveDuration">00:00:50</str> </lst> </requestHandler>
The problem appeared to be with httpclient.
I turned on debug logging for all libraries and saw a message "Garbage in response" coming from httpclient just before the failure.
this is a log snippet:
31 Oct 2013 18:10:40,360 [explicit-fetchindex-cmd] DEBUG DefaultClientConnection - Sending request: GET /solr-master/replication?comman
d=filecontent&generation=6814&qt=%2Freplication&file=_aa7_Lucene41_0.pos&checksum=true&wt=filestream HTTP/1.1
31 Oct 2013 18:10:40,361 [explicit-fetchindex-cmd] DEBUG wire - >> "GET /solr-master/replication?command=filecontent&generation=6814&qt
=%2Freplication&file=_aa7_Lucene41_0.pos&checksum=true&wt=filestream HTTP/1.1[\r][\n]"
31 Oct 2013 18:10:40,361 [explicit-fetchindex-cmd] DEBUG wire - >> "User-Agent: Solr[org.apache.solr.client.solrj.impl.HttpSolrServer]
1.0[\r][\n]"
31 Oct 2013 18:10:40,361 [explicit-fetchindex-cmd] DEBUG wire - >> "Host: solr-master.saltdev.sealdoc.com:8081[\r][\n]"
31 Oct 2013 18:10:40,361 [explicit-fetchindex-cmd] DEBUG wire - >> "Connection: Keep-Alive[\r][\n]"
31 Oct 2013 18:10:40,361 [explicit-fetchindex-cmd] DEBUG wire - >> "[\r][\n]"
31 Oct 2013 18:10:40,361 [explicit-fetchindex-cmd] DEBUG headers - >> GET /solr-master/replication?command=filecontent&generation=6814&
qt=%2Freplication&file=_aa7_Lucene41_0.pos&checksum=true&wt=filestream HTTP/1.1
31 Oct 2013 18:10:40,361 [explicit-fetchindex-cmd] DEBUG headers - >> User-Agent: Solr[org.apache.solr.client.solrj.impl.HttpSolrServer
] 1.0
31 Oct 2013 18:10:40,361 [explicit-fetchindex-cmd] DEBUG headers - >> Host: solr-master.saltdev.sealdoc.com:8081
31 Oct 2013 18:10:40,361 [explicit-fetchindex-cmd] DEBUG headers - >> Connection: Keep-Alive
31 Oct 2013 18:10:40,361 [explicit-fetchindex-cmd] DEBUG wire - << "[\r][\n]"
31 Oct 2013 18:10:40,361 [explicit-fetchindex-cmd] DEBUG DefaultHttpResponseParser - Garbage in response:
31 Oct 2013 18:10:40,361 [explicit-fetchindex-cmd] DEBUG wire - << "4[\r][\n]"
31 Oct 2013 18:10:40,361 [explicit-fetchindex-cmd] DEBUG DefaultHttpResponseParser - Garbage in response: 4
31 Oct 2013 18:10:40,361 [explicit-fetchindex-cmd] DEBUG wire - << "[0x0][0x0][0x0][0x0][\r][\n]"
31 Oct 2013 18:10:40,361 [explicit-fetchindex-cmd] DEBUG DefaultHttpResponseParser - Garbage in response: ^#^#^#^#
31 Oct 2013 18:10:40,361 [explicit-fetchindex-cmd] DEBUG wire - << "0[\r][\n]"
31 Oct 2013 18:10:40,361 [explicit-fetchindex-cmd] DEBUG DefaultHttpResponseParser - Garbage in response: 0
31 Oct 2013 18:10:40,361 [explicit-fetchindex-cmd] DEBUG wire - << "[\r][\n]"
31 Oct 2013 18:10:40,361 [explicit-fetchindex-cmd] DEBUG DefaultHttpResponseParser - Garbage in response:
31 Oct 2013 18:10:40,398 [explicit-fetchindex-cmd] DEBUG DefaultClientConnection - Connection 0.0.0.0:55266<->172.16.77.121:8081 closed
31 Oct 2013 18:10:40,398 [explicit-fetchindex-cmd] DEBUG DefaultClientConnection - Connection 0.0.0.0:55266<->172.16.77.121:8081 shut down
31 Oct 2013 18:10:40,398 [explicit-fetchindex-cmd] DEBUG DefaultClientConnection - Connection 0.0.0.0:55266<->172.16.77.121:8081 closed
31 Oct 2013 18:10:40,398 [explicit-fetchindex-cmd] DEBUG PoolingClientConnectionManager - Connection released: [id: 0][route: {}->http://solr-master.saltdev.sealdoc.com:8081][total kept alive: 1; route allocated: 1 of 10000; total allocated: 1 of 10000]
31 Oct 2013 18:10:40,425 [explicit-fetchindex-cmd] DEBUG CachingDirectoryFactory - Releasing directory: /opt/watchdox/solr-slave/data/index 2 false
31 Oct 2013 18:10:40,425 [explicit-fetchindex-cmd] DEBUG CachingDirectoryFactory - Reusing cached directory: CachedDir<>
31 Oct 2013 18:10:40,425 [explicit-fetchindex-cmd] DEBUG CachingDirectoryFactory - Releasing directory: /opt/watchdox/solr-slave/data 0 false
31 Oct 2013 18:10:40,425 [explicit-fetchindex-cmd] DEBUG CachingDirectoryFactory - Reusing cached directory: CachedDir<>
31 Oct 2013 18:10:40,427 [explicit-fetchindex-cmd] DEBUG CachingDirectoryFactory - Releasing directory: /opt/watchdox/solr-slave/data 0 false
31 Oct 2013 18:10:40,428 [explicit-fetchindex-cmd] DEBUG CachingDirectoryFactory - Done with dir: CachedDir<>
31 Oct 2013 18:10:40,428 [explicit-fetchindex-cmd] DEBUG CachingDirectoryFactory - Releasing directory: /opt/watchdox/solr-slave/data/index.20131031180837277 0 true
31 Oct 2013 18:10:40,428 [explicit-fetchindex-cmd] INFO CachingDirectoryFactory - looking to close /opt/watchdox/solr-slave/data/index.20131031180837277 [CachedDir<>]
31 Oct 2013 18:10:40,428 [explicit-fetchindex-cmd] INFO CachingDirectoryFactory - Closing directory: /opt/watchdox/solr-slave/data/index.20131031180837277
31 Oct 2013 18:10:40,428 [explicit-fetchindex-cmd] INFO CachingDirectoryFactory - Removing directory before core close: /opt/watchdox/solr-slave/data/index.20131031180837277
31 Oct 2013 18:10:40,878 [explicit-fetchindex-cmd] DEBUG CachingDirectoryFactory - Removing from cache: CachedDir<>
31 Oct 2013 18:10:40,878 [explicit-fetchindex-cmd] DEBUG CachingDirectoryFactory - Releasing directory: /opt/watchdox/solr-slave/data/index 1 false
31 Oct 2013 18:10:40,879 [explicit-fetchindex-cmd] ERROR ReplicationHandler - SnapPull failed :org.apache.solr.common.SolrException: Unable to download _aa7_Lucene41_0.pos completely. Downloaded 0!=1081710
at org.apache.solr.handler.SnapPuller$DirectoryFileFetcher.cleanup(SnapPuller.java:1212)
at org.apache.solr.handler.SnapPuller$DirectoryFileFetcher.fetchFile(SnapPuller.java:1092)
at org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:719)
at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:397)
at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:317)
at org.apache.solr.handler.ReplicationHandler$1.run(ReplicationHandler.java:218)
31 Oct 2013 18:10:40,910 [http-bio-8080-exec-8] DEBUG CachingDirectoryFactory - Reusing cached directory: CachedDir<>
So I upgraded the httpcomponents jars to their latest 4.3.x version and the problem disappeared.
the httpcomponents jars which are dependencies of solrj where in the 4.2.x version, I upgraded to httpclient-4.3.1 , httpcore-4.3 and httpmime-4.3.1
I ran the replication a few times now and no problem at all, it is now working as expected.
It seams that the upgrade is necessary only on the slave side but I'm going to upgrade the master too.
How to fix java RuntimeException: Can't find resource 'solrconfig.xml' in classpath?
I'm having issues setting up Solr as Tomcat service in CentOS. I’m using solr 3.3.0 which should be working with Tomcat 5.5 installed from cPanel. The error I was getting is: Can't find resource 'solrconfig.xml' in classpath or '/home/sampleuser/public_html/apache-solr/example/solr/./conf/', cwd=/usr/local/jakarta/apache-tomcat-5.5.33/bin I saw other people having similar issues. I googled and tried to apply all of the approaches. However nothing worked for me so far. I did the following: Copied solr.war to tomcat/webapps Restarted Tomcat Service Edited /usr/local/jakarta/tomcat/webapps/solr/WEB-INF/web.xml as: <env-entry> <env-entry-name>solr/home</env-entry-name> <env-entry-value>/home/sampleuser/public_html/apache-solr/example/solr</env-entry-value> <env-entry-type>java.lang.String</env-entry-type> </env-entry> Created solr.xml in /usr/local/jakarta/tomcat/conf/Catalina/localhost as: <?xml version="1.0" encoding="utf-8"?> <Context docBase="/usr/local/jakarta/tomcat/webapps/solr.war" debug="0" crossContext="true"> <Environment name="solr/home" type="java.lang.String" value="/home/sampleuser/public_html/apache-solr/example"override="true"/> </Context> Edited tomcat startup script at etc/init.d (not sure if I had to do it at all) with: export JAVA_OPTS="$JAVA_OPTS -Dsolr.solr.home=/home/sampleuser/public_html/apache-solr/example/solr" Below is the complete error log from catalina.err: Oct 9, 2011 7:35:06 PM org.apache.catalina.core.StandardService stop INFO: Stopping service Catalina Oct 9, 2011 7:35:06 PM org.apache.catalina.core.ApplicationContext log INFO: SessionListener: contextDestroyed() Oct 9, 2011 7:35:06 PM org.apache.catalina.core.ApplicationContext log INFO: ContextListener: contextDestroyed() Oct 9, 2011 7:35:06 PM org.apache.catalina.core.ApplicationContext log INFO: SessionListener: contextDestroyed() Oct 9, 2011 7:35:06 PM org.apache.catalina.core.ApplicationContext log INFO: ContextListener: contextDestroyed() Oct 9, 2011 7:35:06 PM org.apache.coyote.http11.Http11BaseProtocol destroy INFO: Stopping Coyote HTTP/1.1 on http-8080 Oct 9, 2011 7:35:21 PM org.apache.catalina.core.AprLifecycleListener init INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/local/jdk/jre/lib/amd64/server:/usr/local/jdk/jre/lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib Oct 9, 2011 7:35:23 PM org.apache.coyote.http11.Http11BaseProtocol init INFO: Initializing Coyote HTTP/1.1 on http-8080 Oct 9, 2011 7:35:23 PM org.apache.catalina.startup.Catalina load INFO: Initialization processed in 2875 ms Oct 9, 2011 7:35:23 PM org.apache.catalina.core.StandardService start INFO: Starting service Catalina Oct 9, 2011 7:35:23 PM org.apache.catalina.core.StandardEngine start INFO: Starting Servlet Engine: Apache Tomcat/5.5.33 Oct 9, 2011 7:35:23 PM org.apache.catalina.core.StandardHost start INFO: XML validation disabled Oct 9, 2011 7:35:24 PM org.apache.catalina.startup.HostConfig deployDescriptor WARNING: A docBase /usr/local/jakarta/tomcat/webapps/solr.war inside the host appBase has been specified, and will be ignored Oct 9, 2011 7:35:25 PM org.apache.solr.core.SolrResourceLoader locateSolrHome INFO: Using JNDI solr.home: /home/sampleuser/public_html/apache-solr/example/solr Oct 9, 2011 7:35:25 PM org.apache.solr.core.SolrResourceLoader <init> INFO: Solr home set to '/home/sampleuser/public_html/apache-solr/example/solr/' Oct 9, 2011 7:35:25 PM org.apache.solr.servlet.SolrDispatchFilter init INFO: SolrDispatchFilter.init() Oct 9, 2011 7:35:25 PM org.apache.solr.core.SolrResourceLoader locateSolrHome INFO: Using JNDI solr.home: /home/sampleuser/public_html/apache-solr/example/solr Oct 9, 2011 7:35:25 PM org.apache.solr.core.CoreContainer$Initializer initialize INFO: looking for solr.xml: /home/sampleuser/public_html/apache-solr/example/solr/solr.xml Oct 9, 2011 7:35:25 PM org.apache.solr.core.SolrResourceLoader locateSolrHome INFO: Using JNDI solr.home: /home/sampleuser/public_html/apache-solr/example/solr Oct 9, 2011 7:35:25 PM org.apache.solr.core.CoreContainer <init> INFO: New CoreContainer: solrHome=/home/sampleuser/public_html/apache-solr/example/solr/ instance=1537767228 Oct 9, 2011 7:35:25 PM org.apache.solr.core.SolrResourceLoader <init> INFO: Solr home set to '/home/sampleuser/public_html/apache-solr/example/solr/' Oct 9, 2011 7:35:26 PM org.apache.solr.core.SolrResourceLoader <init> INFO: Solr home set to '/home/sampleuser/public_html/apache-solr/example/solr/./' Oct 9, 2011 7:35:26 PM org.apache.solr.common.SolrException log SEVERE: java.lang.RuntimeException: Can't find resource 'solrconfig.xml' in classpath or '/home/sampleuser/public_html/apache-solr/example/solr/./conf/', cwd=/usr/local/jakarta/apache-tomcat-5.5.33/bin at org.apache.solr.core.SolrResourceLoader.openResource(SolrResourceLoader.java:268) at org.apache.solr.core.SolrResourceLoader.openConfig(SolrResourceLoader.java:234) at org.apache.solr.core.Config.<init>(Config.java:141) at org.apache.solr.core.SolrConfig.<init>(SolrConfig.java:131) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:435) at org.apache.solr.core.CoreContainer.load(CoreContainer.java:316) at org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:133) at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:94) at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:221) at org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:302) at org.apache.catalina.core.ApplicationFilterConfig.<init>(ApplicationFilterConfig.java:78) at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:3666) at org.apache.catalina.core.StandardContext.start(StandardContext.java:4258) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:760) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:740) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:544) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:634) at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:561) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:496) at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1203) at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:319) at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:120) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1022) at org.apache.catalina.core.StandardHost.start(StandardHost.java:736) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1014) at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443) at org.apache.catalina.core.StandardService.start(StandardService.java:448) at org.apache.catalina.core.StandardServer.start(StandardServer.java:700) at org.apache.catalina.startup.Catalina.start(Catalina.java:552) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:295) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:219) Oct 9, 2011 7:35:26 PM org.apache.solr.servlet.SolrDispatchFilter init INFO: user.dir=/usr/local/jakarta/apache-tomcat-5.5.33/bin Oct 9, 2011 7:35:26 PM org.apache.solr.servlet.SolrDispatchFilter init INFO: SolrDispatchFilter.init() done Oct 9, 2011 7:35:26 PM org.apache.solr.servlet.SolrServlet init INFO: SolrServlet.init() Oct 9, 2011 7:35:26 PM org.apache.solr.core.SolrResourceLoader locateSolrHome INFO: Using JNDI solr.home: /home/sampleuser/public_html/apache-solr/example/solr Oct 9, 2011 7:35:26 PM org.apache.solr.servlet.SolrServlet init INFO: SolrServlet.init() done Oct 9, 2011 7:35:26 PM org.apache.solr.core.SolrResourceLoader locateSolrHome INFO: Using JNDI solr.home: /home/sampleuser/public_html/apache-solr/example/solr Oct 9, 2011 7:35:26 PM org.apache.solr.servlet.SolrUpdateServlet init INFO: SolrUpdateServlet.init() done When I copy the missing path via shell I was able to navigate just fine. Seem like Tomcat cannot locate the solrconfig.xml.
First off: you don't need to #3 and #4 and 5. #3 isn't really recommended unless you are explicitly customizing the solr.war. #4 and #5 are two different ways to do the same thing - use whichever one is most convenient for you (JNDI vs system properties) Second: please note that while #3 and #5 are consistent, in #4 you actually specify a different Solr home dir (the parent dir of what you really want) Third: I think the crux of your problem is file permissions. The logging is a bit confusing due to SOLR-3264 which causes "instanceDirs" for each core to be refered to erroneously as "Solr Home" but what seems to be happening is that it can't find (or read) your "/home/sampleuser/public_html/apache-solr/example/solr/solr.xml" so it's going into legacy "Single Core" mode where it assumes that you want to use your Solr Home dir as an instanceDir for a (single) Solr Core -- so it then looks for "/home/sampleuser/public_html/apache-solr/example/solr/solrconf.xml" and if it can't find (or read) that file it gives up. So depending on whether your main concern is that you actually have a solr.xml (pointing to alternate instanceDirs) hat it's not finding, or if you actually have a solrconfig.xml file where it's looking, check the file permissions of those files (and their directories), and make sure they are readable by the effective user of your tomcat server. (FWIW: Solr 3.6 and up will have better logging when solr.xml is not found and legacy mode is used, and I've opened SOLR-3266 to pursue better ways of logging/erroring on unreadable files in the future)