sftp in apache camel tries kerberos authentication - apache-camel

My camel route picks up files from a sftp endpoint configured with username and password. Before the connection is made, i get a prompt for username and password from Kerberos.
I skip this by pressing enter, and authentication proceeds as expected, the files i picked up. Edit: After going through the versioning logs and the program logs i'm not entirely sure this is what happend, and i cannot reproduce it.
What i see it that my (one) screen-copy with the Kerberos prompt coincides with the one of several logs showing authentication failure because my username is wrong.
Why do i get the prompt? My endpoint looks like this:
sendEmptyMessageWhenIdle: sendEmptyMessageWhenIdle=true
useHostFile: knownHostsFile=known_hosts&strictHostKeyChecking=yes
workDirectory: /omg/myprogram/${env}/WorkDirectory
fromParams: streamDownload=true&maximumReconnectAttempts=2&throwExceptionOnConnectFailed=true&consumer.bridgeErrorHandler=true&disconnectOnBatchComplete=true
mypassword: password=blablabla
fromUri: sftp://myuser#accept.server.nu:22/TST?${mypassword}
from: ${fromUri}&${sendEmptyMessageWhenIdle}&${routeScheduler}&delete=true&${useHostFile}&localWorkDirectory=${workDirectory}&${fromParams}
Actually when i dont use a hosts file the same thing happens:
useUserKnownHostsFile=false
The camel version is 2.22.2.
What can i do to disable Kerberos in Apache camel?

Could you please set preferredAuthentications attribute on the SFTP component and see if makes a difference? This is an example preferredAuthentications=publickey,keyboard-interactive,password. Documentation for FTP component here. The docs say if you don't set this, the component will use default list of mechanisms in JSCH.
Edit:
I see a similar question here but in JSCH perspective.

It appears that the Kerberos prompt appears when authentication with the provided user-name and password fails.
That is, i have not been able to reproduce the problem i described above but i have found that the Kerberos prompt appeared when the username was incorrect.
Log:
2019-01-31 12:42:30,767 DEBUG [main] myprogram.StartUpVerneProphet - - - Running with Spring Boot v2.0.8.RELEASE, Spring v5.0.12.RELEASE
2019-01-31 12:42:30,769 INFO [main] myprogram.StartUpVerneProphet - - - No active profile set, falling back to default profiles: default
2019-01-31 12:42:36,003 WARN [main] org.apache.camel.component.file.remote.SftpOperations - - - JSCH -> Permanently added 'accept.server.nu' (DSA) to the list of known hosts.
2019-01-31 12:43:01,537 WARN [main] org.apache.camel.component.file.remote.SftpOperations - - - JSCH -> Permanently added 'accept.server.nu' (DSA) to the list of known hosts.
2019-01-31 12:44:13,940 WARN [main] org.apache.camel.component.file.remote.SftpOperations - - - JSCH -> Permanently added 'accept.server.nu' (DSA) to the list of known hosts.
2019-01-31 12:44:24,963 WARN [main] org.apache.camel.component.file.remote.SftpConsumer - - - Error auto creating directory: TST due Cannot connect to sftp://faulty-username#accept.server.nu:22. This exception is ignored. org.apache.camel.component.file.GenericFileOperationFailedException: Cannot connect to sftp://faulty-username#accept.server.nu:22
at org.apache.camel.component.file.remote.SftpOperations.connect(SftpOperations.java:144)
at org.apache.camel.component.file.remote.RemoteFileConsumer.connectIfNecessary(RemoteFileConsumer.java:197)
Screen-copy:
12:42:30,761 |-INFO in c.q.l.core.rolling.helper.TimeBasedArchiveRemover - first clean up after appender initialization
12:42:30,763 |-INFO in c.q.l.core.rolling.helper.TimeBasedArchiveRemover - Multiple periods, i.e. 32 periods, seem to have elapsed. This is expected a t application start.
2019-01-31 12:42:30 DEBUG myprogram.StartUp - - - Running with Spring Boot v2.0.8.RELEASE, Spr ing v5.0.12.RELEASE
2019-01-31 12:42:30 INFO myprogram.StartUp - - - No active profile set, falling back to defau lt profiles: default
2019-01-31 12:42:36 WARN org.apache.camel.component.file.remote.SftpOperations - - - JSCH -> Permanently added 'accept.server.nu' (DSA) to the list of known hosts.
Kerberos username [root]:
Kerberos password for root:
2019-01-31 12:43:01 WARN org.apache.camel.component.file.remote.SftpOperations - - - JSCH -> Permanently added 'accept.server.nu' (DSA) to the list of known hosts.
Kerberos username [root]:
Kerberos password for root:
Unfortunately, i made only one screencopy, so i am unsure if there was any occasion where the Kerberos prompt appeared after i used the correct user-name.
It hasn't appeared totday.

Related

gatling-3.0.0: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate

i am using Gatling 3.0.0 as a plugin in SBT i am configuring the browser as given in the https://gatling.io/docs/current/http/recorder/#recorder under configuration heading after then when i start the recorder using gatling:startRecorder in sbt and try to hit my website https://www.example.com/ Firefox displayed
Did Not Connect: Potential Security Issue
Firefox detected a potential security threat and did not continue to www.mydomain.com because this website requires a secure connection.
www.mydomain.com has a security policy called HTTP Strict Transport Security (HSTS), which means that Firefox can only connect to it securely. You can’t add an exception to visit this site
and here are the exception logs
ioEventLoopGroup-2-1] DEBUG io.netty.handler.ssl.util.InsecureTrustManagerFactory - Accepting a server certificate: CN=www.mydomain.com
14:44:55.604 [nioEventLoopGroup-4-2] DEBUG io.gatling.recorder.http.Mitm$ - Open new server channel
14:44:55.607 [nioEventLoopGroup-4-1] WARN io.netty.channel.DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:128)
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:308)
at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:279)
at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:181)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164)
at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:672)
at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:627)
at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:443)
at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:422)
at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:634)
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:294)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1297)
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1199)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1243)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
... 16 common frames omitted
14:44:55.609 [recorder-akka.actor.default-dispatcher-4] DEBUG io.gatling.recorder.http.flows.SecuredNoProxyMitmActor - Server channel 6acf48e4 was closed while in Connected state, closing
14:44:55.622 [recorder-akka.actor.default-dispatcher-2] DEBUG io.gatling.recorder.http.flows.PlainNoProxyMitmActor - serverChannel=8d7b2171 received init request http://detectportal.firefox.com/success.txt, connecting
14:44:55.622 [recorder-akka.actor.default-dispatcher-2] DEBUG io.gatling.recorder.http.flows.PlainNoProxyMitmActor - Connecting to Remote(detectportal.firefox.com,80)
14:44:55.629 [recorder-akka.actor.default-dispatcher-4] INFO akka.actor.RepointableActorRef - Message [io.gatling.recorder.http.flows.MitmMessage$ClientChannelInactive] without sender to Actor[akka://recorder/user/$a#-1754914561] was not delivered. [1] dead letters encountered. If this is not an expected behavior, then [Actor[akka://recorder/user/$a#-1754914561]] may have terminated unexpectedly, This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
14:44:55.655 [nioEventLoopGroup-2-2] DEBUG io.gatling.recorder.http.Mitm$ - Open new client channel
How did you configured proxy in your firefox? Do you have proxy only for http or also for https? If you are proxing also https requests then in recorder settings you need to switch "HTTPS mode" to "Certificate Authority". There will be a button to generate new certificate authority file that you need to import to your browser (Preferences / Privacy & Security / Certificates / View certificates / Import). After that your browser will know that it can trust Gatling proxy server and you should be able to proxy also ssl requests.

Replace zookeeper server from zookeeper ensemble (with SolrCloud)

I have a SolrCloud cluster (6.6) setup with external Zookeeper Ensemble (3.4.8) of 5 nodes. Recently, one machine (ip1:port1) that run 1 Zookeeper with id=1 went down. This is what I've done to replace zookeeper:
Start zookeeper in another machine with the same id (=1).
Change zoo.cfg in 4 live zookeeper to match new zookeeper server and restart.
Update ZK_HOST variable in solr.in.sh to match new zookeeper server.
Restart solr.
After that, my solr cluster seemed to functioning well, but in solr.log, it looked like solr client and zookeeper servers still try to connect to the old zookeeper:
Solr log
2017-12-01 15:04:38.782 WARN (Timer-0-SendThread(ip1:port1)) [ ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 30029ms for sessionid 0x0
2017-12-01 15:04:40.807 WARN (Timer-0-SendThread(ip1:port1)) [ ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 31030ms for sessionid 0x0
Zookeeper log:
2017-12-01 13:53:57,972 [myid:] - INFO [main-SendThread(ip1:port1):ClientCnxn$SendThread#1032] - Opening socket connection to server ip1:port1. Will not attempt to authenticate using SASL (unknown error)
2017-12-01 13:54:03,972 [myid:] - WARN [main-SendThread(ip1:port1):ClientCnxn$SendThread#1162] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
2017-12-01 13:54:05,074 [myid:] - INFO [main-SendThread(ip1:port1):ClientCnxn$SendThread#1032] - Opening socket connection to server ip1:port1. Will not attempt to authenticate using SASL (unknown error)
2017-12-01 13:54:06,974 [myid:] - WARN [main-SendThread(ip1:port1):ClientCnxn$SendThread#1162] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
I've done some search in add/remove zookeeper but didn't find a document for it. My zookeeper version (3.4.7) is not supported for dynamic reconfiguration (which is in zookeeper 3.5).
Is there a way I can manually remove/add zookeeper server from ensemble?
Thanks for your attention!

WSO2 DSS Query Result Inconsistent

We have a blocking issue trying to execute queries via WSO2 DSS.
After a day of usage, the query results are not being retrieved by DSS consistently. (The same query might return results in the next try)
Using a network monitoring tool, we could find the DB responds almost immediately, however DSS did not acknowledge this response.
The same code in Acceptance env did not have this problem.
This is what i get in my log :
[2015-11-27 09:45:17,308] WARN - SourceHandler Connection time out after request is read: http-incoming-1366
[2015-11-27 09:45:17,314] WARN - TargetHandler http-outgoing-1267: Connection time out while in state: REQUEST_DONE
[2015-11-27 09:45:17,315] WARN - FaultHandler ERROR_CODE : 101507
[2015-11-27 09:45:17,315] WARN - FaultHandler ERROR_MESSAGE : Error in Sender
[2015-11-27 09:45:17,315] WARN - FaultHandler ERROR_DETAIL : Error in Sender
[2015-11-27 09:45:17,316] WARN - FaultHandler ERROR_EXCEPTION : null
[2015-11-27 09:45:17,316] WARN - FaultHandler FaultHandler : Endpoint [Sample_First]
[2015-11-27 09:45:17,316] WARN - FailoverEndpoint Endpoint [getMyOverviewServiceEndpoint] Detect a Failure in a child endpoint : Endpoint [Sample_First]
Any pointers could be of great help!
Thanks,
Siby Mathew
Update
Could find that WSO2 initially connects to Google DNS in the production, gets timed out and then connect to proper DNS to resolve the Database server.
Could this be the reason for the inconsistent behaviour ?
Is there any default code in DSS to connect to Google DNS ?

Deploy app in google app engine

while deplying app in google app engine i always got following errors...
2014-08-10 03:30:58 Running command: "['C:\\Python27\\python.exe', '-u', u'C:\\Program Files (x86)\\Google\\google_appengine\\appcfg.py', '--no_cookies', u'--email=rjt.rjtshrm94#gmail.com', '--passin', 'update', u'C:\\Users\\Rjt\\Desktop\\New folder (3)']"
03:31 AM Application: my-app-chat; version: 1
03:31 AM Host: appengine.google.com
03:31 AM
Starting update of app: my-app-chat, version: 1
03:31 AM Getting current resource limits.
Password for rjt.rjtshrm94#gmail.com: Invalid username or password.
2014-08-10 03:31:08,612 ERROR appcfg.py:2415 An error occurred processing file '': HTTP Error 401: Unauthorized. Aborting.
Error 401: --- begin server output ---
Must authenticate first.
--- end server output ---
2014-08-10 03:31:08 (Process exited with code 1)
but my emailid and password are correct....
You can close this window now.
I had the same problem and it was driving me mad. I wasn't typing the password wrong.
For me, changing a security setting, not in the app but in my Google account solved it. Go here: https://www.google.com/settings/security/lesssecureapps and "Enable" less secure devices and apps to access your data. For me, that fixed it.
Otherwise I've come accross time settings being the problem (mentioned above).
But more importantly, double-check that the name of your app, that you registered on GoogleDev, is the name that you put in the app.yaml. If it is, then hitting the dashboard in the Launcher should take you to the dashboard of your app.
See my question and answer for more information, if none of this works.

zookeeper is not running after restart

I have 3 zookeeper nodes. Those node was working fine but when I restart those nodes using ./zkServer.sh restart, the zookeeper did not got up again.
When I checked on the zookeeper status, it return:
./zkServer.sh status
JMX enabled by default
Using config: /opt/zookeeper/bin/../conf/zoo.cfg
Error contacting service. It is probably not running.
my zoo.cnf is:
dataDir=/var/lib/zookeeperdata/3
clientPort=2181
initLimit=50
tickTime=2000
syncLimit=10
maxClientCnxns=100000
server.1=IP1 value:2888:3888
server.2=IP2 value:2889:3889
server.3=127.0.0.1:2890:3890
This is unstable behavior because may be after two hours or tomorrow if I made restart for the 3 zookeeper nodes, they will see each others and working fine because this happened before with me.
zookeeper log:
2014-05-14 15:22:34,236 [myid:3] - INFO [main:NIOServerCnxnFactory#94] - binding to port 0.0.0.0/0.0.0.0:2181
2014-05-14 15:22:34,282 [myid:3] - INFO [main:QuorumPeer#913] - tickTime set to 2000
2014-05-14 15:22:34,283 [myid:3] - INFO [main:QuorumPeer#933] - minSessionTimeout set to -1
2014-05-14 15:22:34,283 [myid:3] - INFO [main:QuorumPeer#944] - maxSessionTimeout set to -1
2014-05-14 15:22:34,283 [myid:3] - INFO [main:QuorumPeer#959] - initLimit set to 50
2014-05-14 15:22:34,356 [myid:3] - INFO [main:FileSnap#83] - Reading snapshot /var/lib/zookeeperdata/3/version-2/snapshot.f100000001
2014-05-14 15:22:43,387 [myid:3] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory#197] - Accepted socket connection from /127.0.0.1:50923
2014-05-14 15:22:43,396 [myid:3] - INFO [Thread-1:QuorumCnxManager$Listener#486] - My election bind port: 0.0.0.0/0.0.0.0:3890
2014-05-14 15:22:43,404 [myid:3] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn#354] - Exception causing close of session 0x0 due to java.io.IOExce
ption: ZooKeeperServer not running
2014-05-14 15:22:43,404 [myid:3] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn#1001] - Closed socket connection for client /127.0.0.1:50923 (no se
ssion established for client)
2014-05-14 15:22:43,427 [myid:3] - INFO [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:2181:QuorumPeer#670] - LOOKING
2014-05-14 15:22:43,429 [myid:3] - INFO [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#740] - New election. My id = 3, proposed zxid=0xf100000001
2014-05-14 15:22:48,438 [myid:3] - WARN [WorkerSender[myid=3]:QuorumCnxManager#368] - Cannot open channel to 1 at election address /54.76.10.81:3888
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:354)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:327)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:393)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:365)
at java.lang.Thread.run(Thread.java:662)
2014-05-14 15:22:53,440 [myid:3] - WARN [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager#368] - Cannot open channel to 1 at election address /54.76.10.81:3
888
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:354)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:388)
I searched a lot on this but I did not found anything useful for me so I hope someone can help me.
Thanks
I've seen behavior like this as well. A ZK configuration that's been running fine will sometimes simply fail to restart. When this happens I've tried the following:
1) look at the logs for all of the servers...often one will list an error
2) stop all servers and restart
3) stop all servers and restart the servers one at a time
4) verify that each server's myid file exists, has correct permissions and has the right value.
I've used clusterssh to open windows to each of the servers so that the restarts can be at the very same time...and then I've tailed all of the server logs. Keep in mind that during restart the ZK cluster is doing a lot: both starting each server and electing a leader. I've had times when the cluster seemed to fail and then after a few more minutes it seems to figure it out.
There is a great tool called zktop that I've used for monitoring ZK.
I fixed it by changing the IP 127.0.0.1 to the internal IP for amazon node, after making this change for the three nodes and restart, this problem did not happened again. I hope this answer can help someone asking about the same problem.
make sure you have put correct data Dir in each of your node configuration.
and also put a myid file in data Dir and put a number between 1-255 for each of you node in the myid file.
I think it resole the issue.

Resources