We have a blocking issue trying to execute queries via WSO2 DSS.
After a day of usage, the query results are not being retrieved by DSS consistently. (The same query might return results in the next try)
Using a network monitoring tool, we could find the DB responds almost immediately, however DSS did not acknowledge this response.
The same code in Acceptance env did not have this problem.
This is what i get in my log :
[2015-11-27 09:45:17,308] WARN - SourceHandler Connection time out after request is read: http-incoming-1366
[2015-11-27 09:45:17,314] WARN - TargetHandler http-outgoing-1267: Connection time out while in state: REQUEST_DONE
[2015-11-27 09:45:17,315] WARN - FaultHandler ERROR_CODE : 101507
[2015-11-27 09:45:17,315] WARN - FaultHandler ERROR_MESSAGE : Error in Sender
[2015-11-27 09:45:17,315] WARN - FaultHandler ERROR_DETAIL : Error in Sender
[2015-11-27 09:45:17,316] WARN - FaultHandler ERROR_EXCEPTION : null
[2015-11-27 09:45:17,316] WARN - FaultHandler FaultHandler : Endpoint [Sample_First]
[2015-11-27 09:45:17,316] WARN - FailoverEndpoint Endpoint [getMyOverviewServiceEndpoint] Detect a Failure in a child endpoint : Endpoint [Sample_First]
Any pointers could be of great help!
Thanks,
Siby Mathew
Update
Could find that WSO2 initially connects to Google DNS in the production, gets timed out and then connect to proper DNS to resolve the Database server.
Could this be the reason for the inconsistent behaviour ?
Is there any default code in DSS to connect to Google DNS ?
Related
I am calling a function from a smart contract:
ContractCallQuery query = new ContractCallQuery()
.setContractId(ContractId.fromString(request.getContractID()))
.setGas(100_000_000)
.setFunction(request.getContractFunctionName(), contractFunctionParameters);
Which results in a BUSY response (timing out at attempt 47):
web - 2023-01-30 22:53:28,795 [Test worker] WARN c.h.h.sdk.Query$QueryCostQuery - Problem submitting request to node 0.0.3 for attempt #43, retry with new node: BUSY
web - 2023-01-30 22:53:28,936 [Test worker] WARN c.h.h.sdk.Query$QueryCostQuery - Retrying node 0.0.6 in 8000 ms after failure during attempt #44: BUSY
web - 2023-01-30 22:53:37,032 [Test worker] WARN c.h.h.sdk.Query$QueryCostQuery - Problem submitting request to node 0.0.5 for attempt #45, retry with new node: BUSY
web - 2023-01-30 22:53:37,182 [Test worker] WARN c.h.h.sdk.Query$QueryCostQuery - Problem submitting request to node 0.0.3 for attempt #46, retry with new node: BUSY
web - 2023-01-30 22:53:37,321 [Test worker] WARN c.h.h.sdk.Query$QueryCostQuery - Retrying node 0.0.6 in 8000 ms after failure during attempt #47: BUSY
java.util.concurrent.TimeoutException
This used to work and I havent chaged any code. How do I resolve this?
The BUSY error happens when the network gets congested with other users and the throttle has reached its limit. You'll have to wait until the network is no longer congested to complete any transactions.
If you are on testnet consider running a local instance to continue testing. Its highly encouraged to use a local node for development.
Here is a tutorial on how to set up your own local node. https://hedera.com/blog/how-to-set-up-your-own-hedera-local-network-using-docker
We are facing this below error in Vespa, after restarting the cluster we got this below issue.
1600455444.680758 10.10.000.00 1030/1 container Container.com.yahoo.filedistribution.fileacquirer.FileAcquirerImpl info Retrying waitFor for file 'e0ce64d459828eb0': 103 -- Request timed out after 60.0 seconds.
1600455446.819853 10.10.000.00 32752/146 configproxy configproxy.com.yahoo.vespa.filedistribution.FileReferenceDownloader info Request failed. Req: request filedistribution.serveFile(e0ce64d459828eb0,0)\nSpec: tcp/10.10.000.00:19070, error code: 103, set error for connection and use another for next request
We faced this issue second time, earlier we kept it ideal and it was resolved automatically, but this time it is persistent.
Looks like the configproxy is unable to talk to the config server (which is listening to port 19070 on the same host: Spec: tcp/10.10.000.00:19070). Is the config server really runnning and listening on port 19070 on this host? Try running the vespa-config-status script to see if all is well with the config system
i am using Gatling 3.0.0 as a plugin in SBT i am configuring the browser as given in the https://gatling.io/docs/current/http/recorder/#recorder under configuration heading after then when i start the recorder using gatling:startRecorder in sbt and try to hit my website https://www.example.com/ Firefox displayed
Did Not Connect: Potential Security Issue
Firefox detected a potential security threat and did not continue to www.mydomain.com because this website requires a secure connection.
www.mydomain.com has a security policy called HTTP Strict Transport Security (HSTS), which means that Firefox can only connect to it securely. You can’t add an exception to visit this site
and here are the exception logs
ioEventLoopGroup-2-1] DEBUG io.netty.handler.ssl.util.InsecureTrustManagerFactory - Accepting a server certificate: CN=www.mydomain.com
14:44:55.604 [nioEventLoopGroup-4-2] DEBUG io.gatling.recorder.http.Mitm$ - Open new server channel
14:44:55.607 [nioEventLoopGroup-4-1] WARN io.netty.channel.DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:128)
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:308)
at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:279)
at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:181)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164)
at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:672)
at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:627)
at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:443)
at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:422)
at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:634)
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:294)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1297)
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1199)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1243)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
... 16 common frames omitted
14:44:55.609 [recorder-akka.actor.default-dispatcher-4] DEBUG io.gatling.recorder.http.flows.SecuredNoProxyMitmActor - Server channel 6acf48e4 was closed while in Connected state, closing
14:44:55.622 [recorder-akka.actor.default-dispatcher-2] DEBUG io.gatling.recorder.http.flows.PlainNoProxyMitmActor - serverChannel=8d7b2171 received init request http://detectportal.firefox.com/success.txt, connecting
14:44:55.622 [recorder-akka.actor.default-dispatcher-2] DEBUG io.gatling.recorder.http.flows.PlainNoProxyMitmActor - Connecting to Remote(detectportal.firefox.com,80)
14:44:55.629 [recorder-akka.actor.default-dispatcher-4] INFO akka.actor.RepointableActorRef - Message [io.gatling.recorder.http.flows.MitmMessage$ClientChannelInactive] without sender to Actor[akka://recorder/user/$a#-1754914561] was not delivered. [1] dead letters encountered. If this is not an expected behavior, then [Actor[akka://recorder/user/$a#-1754914561]] may have terminated unexpectedly, This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
14:44:55.655 [nioEventLoopGroup-2-2] DEBUG io.gatling.recorder.http.Mitm$ - Open new client channel
How did you configured proxy in your firefox? Do you have proxy only for http or also for https? If you are proxing also https requests then in recorder settings you need to switch "HTTPS mode" to "Certificate Authority". There will be a button to generate new certificate authority file that you need to import to your browser (Preferences / Privacy & Security / Certificates / View certificates / Import). After that your browser will know that it can trust Gatling proxy server and you should be able to proxy also ssl requests.
My camel route picks up files from a sftp endpoint configured with username and password. Before the connection is made, i get a prompt for username and password from Kerberos.
I skip this by pressing enter, and authentication proceeds as expected, the files i picked up. Edit: After going through the versioning logs and the program logs i'm not entirely sure this is what happend, and i cannot reproduce it.
What i see it that my (one) screen-copy with the Kerberos prompt coincides with the one of several logs showing authentication failure because my username is wrong.
Why do i get the prompt? My endpoint looks like this:
sendEmptyMessageWhenIdle: sendEmptyMessageWhenIdle=true
useHostFile: knownHostsFile=known_hosts&strictHostKeyChecking=yes
workDirectory: /omg/myprogram/${env}/WorkDirectory
fromParams: streamDownload=true&maximumReconnectAttempts=2&throwExceptionOnConnectFailed=true&consumer.bridgeErrorHandler=true&disconnectOnBatchComplete=true
mypassword: password=blablabla
fromUri: sftp://myuser#accept.server.nu:22/TST?${mypassword}
from: ${fromUri}&${sendEmptyMessageWhenIdle}&${routeScheduler}&delete=true&${useHostFile}&localWorkDirectory=${workDirectory}&${fromParams}
Actually when i dont use a hosts file the same thing happens:
useUserKnownHostsFile=false
The camel version is 2.22.2.
What can i do to disable Kerberos in Apache camel?
Could you please set preferredAuthentications attribute on the SFTP component and see if makes a difference? This is an example preferredAuthentications=publickey,keyboard-interactive,password. Documentation for FTP component here. The docs say if you don't set this, the component will use default list of mechanisms in JSCH.
Edit:
I see a similar question here but in JSCH perspective.
It appears that the Kerberos prompt appears when authentication with the provided user-name and password fails.
That is, i have not been able to reproduce the problem i described above but i have found that the Kerberos prompt appeared when the username was incorrect.
Log:
2019-01-31 12:42:30,767 DEBUG [main] myprogram.StartUpVerneProphet - - - Running with Spring Boot v2.0.8.RELEASE, Spring v5.0.12.RELEASE
2019-01-31 12:42:30,769 INFO [main] myprogram.StartUpVerneProphet - - - No active profile set, falling back to default profiles: default
2019-01-31 12:42:36,003 WARN [main] org.apache.camel.component.file.remote.SftpOperations - - - JSCH -> Permanently added 'accept.server.nu' (DSA) to the list of known hosts.
2019-01-31 12:43:01,537 WARN [main] org.apache.camel.component.file.remote.SftpOperations - - - JSCH -> Permanently added 'accept.server.nu' (DSA) to the list of known hosts.
2019-01-31 12:44:13,940 WARN [main] org.apache.camel.component.file.remote.SftpOperations - - - JSCH -> Permanently added 'accept.server.nu' (DSA) to the list of known hosts.
2019-01-31 12:44:24,963 WARN [main] org.apache.camel.component.file.remote.SftpConsumer - - - Error auto creating directory: TST due Cannot connect to sftp://faulty-username#accept.server.nu:22. This exception is ignored. org.apache.camel.component.file.GenericFileOperationFailedException: Cannot connect to sftp://faulty-username#accept.server.nu:22
at org.apache.camel.component.file.remote.SftpOperations.connect(SftpOperations.java:144)
at org.apache.camel.component.file.remote.RemoteFileConsumer.connectIfNecessary(RemoteFileConsumer.java:197)
Screen-copy:
12:42:30,761 |-INFO in c.q.l.core.rolling.helper.TimeBasedArchiveRemover - first clean up after appender initialization
12:42:30,763 |-INFO in c.q.l.core.rolling.helper.TimeBasedArchiveRemover - Multiple periods, i.e. 32 periods, seem to have elapsed. This is expected a t application start.
2019-01-31 12:42:30 DEBUG myprogram.StartUp - - - Running with Spring Boot v2.0.8.RELEASE, Spr ing v5.0.12.RELEASE
2019-01-31 12:42:30 INFO myprogram.StartUp - - - No active profile set, falling back to defau lt profiles: default
2019-01-31 12:42:36 WARN org.apache.camel.component.file.remote.SftpOperations - - - JSCH -> Permanently added 'accept.server.nu' (DSA) to the list of known hosts.
Kerberos username [root]:
Kerberos password for root:
2019-01-31 12:43:01 WARN org.apache.camel.component.file.remote.SftpOperations - - - JSCH -> Permanently added 'accept.server.nu' (DSA) to the list of known hosts.
Kerberos username [root]:
Kerberos password for root:
Unfortunately, i made only one screencopy, so i am unsure if there was any occasion where the Kerberos prompt appeared after i used the correct user-name.
It hasn't appeared totday.
I am trying to setup zookeeper on ec2 two instances. as given here and here.
I am trying to run zookeeper which fails with an error:
command: bin/zkCli.sh -server localhost:2181
> 2015-03-15 00:22:35,644 [myid:] - INFO [main:ZooKeeper#438] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher#3ff0efca
Welcome to ZooKeeper!
2015-03-15 00:22:35,671 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread#975] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2015-03-15 00:22:35,677 [myid:] - WARN [main-SendThread(localhost:2181):ClientCnxn$SendThread#1102] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
[zk: localhost:2181(CONNECTING) 0] 2015-03-15 00:22:36,796 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread#975] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2015-03-15 00:22:36,797 [myid:] - WARN [main-SendThread(localhost:2181):ClientCnxn$SendThread#1102] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
zoo.cfg as bellow
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/var/lib/zookeeper
clientPort=2181
server.1=localhost:2888:3888
server.2=<My ec2 private IPs>:2889:3889
also I have created myId file as on both ec2 instances - /var/lib/zookeeper/myid
I also tried to edit /ect/hosts file but still facing the same issue.
also how I can start both of the zookeeper instances by 1 command?
Note: Server get started successfully if I tried with bin/zkCli.sh start command.
Thanks in advance!
look zk log zookeeper.out，if there have connection limit error, configure the following to zoo.cfg.
# the maximum number of client connections.
# increase this if you need to handle more clients
maxClientCnxns=60
This is temporary error , for mine after some time , It gone away :-
This is my zoo.conf file ::-
Dir=../data
clientPort=2181
tickTime=2000
initLimit=5
This error occurred when I forgot to run% ZOOKEEPER_HOME% \ bin \ zkserver.cmd
By running, the problem has been resolved.
Correct this property on the server.properties
default would be localhost change it to match the zookeeper server starup ip and port
zookeeper.connect=0.0.0.0:2181