I have configured a sticky session set up with a load balancer (Apache) and three app nodes running Jboss 4.2.2 .
the load balancer uses mod_jk and settings as mentioned in the tutorial here.
http://community.jboss.org/wiki/UsingModjk12WithJBoss;jsessionid=1569CBFB7C3096C59C977CD3F7159A32
I have the jumRoute set as node1 ,node2 and node3 for the three nodes and my workerlist property for load balancer is set as
node1,node2,node3
The tutorial has been followed till the last point but I did not configure the useJK parameters under.the value is still set to false.
The sticky sessions are holding up but I seem to loose session and get this error in my mod_jk log file
[error] ajp_get_reply::jk_ajp_common.c (1926): (node1) Timeout with waiting reply from tomcat. Tomcat is down, stopped or network problems (errno=110)
I personally checked the user logged in on node1 and then moved to node2.
Does Apache redirect to another node when it fails to get reply from node1, How does useJk help in this situation.
---edit 01---
I changed the UseJK value to true but still few users still experience sudden log out which I know due to change in the server node catering the users request.
I also wanted to know whether traffic on the nodes has any affect on sticky session and how to counter it.( I am experiencing high load on all the servers for a few days)
----edit 02 ----
I would also like to know about controlling the number of connections per worker.
controlling the number of ajp connector/connections.
relation between the number of connections of apache load balancer and number of
ajp connections in JBoss worker nodes.
what would be the best configuration between Apache 2.2.3 and JBoss 4.2.2 worker nodes with Tomcat 5.5 connectors.
---- edit03-----
http://community.jboss.org/wiki/OptimalModjk12Configuration
using the above article i just wanted to know the best values for Apache
MaxClients
ThreadPerChild
I found the following note in this article interesting. I haven't tried this, but perhaps could be useful for someone experiencing the same problem.
If you are using mod_jk and have turned sticky sessions on, but your sessions are failing to stick, you have probably failed to set the domain, or you have failed to set the jvmRoute, or you are using a non-standard cookie name to implement the stickyness!
I think in your worker.properties file the workerlist should have loadBalancer worker not the node1,node2 & node3.It should be like this
worker.list=loadmanager
worker.loadmanager.balance_workers=node1,node2,node3
I hope u must have these correct.
Also you have to set UserJK arttribute to set as true for load balancing with sticky session combined with JvmRoute. If set to true, it will insert a JvmRouteFilter
to intercept every request and replace the JvmRoute if it detects a failover.
<attribute name="UseJK">true</attribute>
in deploy/jboss-web.deployer/META-INF/jboss-service.xml
Related
Unable to get through the Hybrid Configuration Wizard in Modern mode. This is necessary because we want to migrate mailboxes. Classic mode works.
It knows that there is a Hybrid Agent, but I can't successfully install with either path of using existing or adding a new one. In Azure there is an App Proxy registration which appears to have the incorrect IP for the route to on-prem. This was due to a misconfiguration of our outgoing firewall. However after the firewall configuration was fixed, the App Proxy still has the old return IP, and there is no way in Azure to remove this record.
I've removed the app proxy components on the server, and let the HCW install again but this record is not updated or removed. Also have gone through 'Classic' path which according to community posts is supposed to remove the App Proxy record, but it doesn't.
According to what I've read, if the record is inactive for 10 days, it will be removed, but I'd rather resolve this without waiting for 10 days.
I've tried patching the record using Graph but it doesn't work.
2022.01.31 22:09:59.707 10333 [Client=UX, fn=SendAsync, Thread=15] FINISH Time=2170.2ms Results=BadRequest {"error":{"code":"InternalUrl_Duplicate","message":"Internal url 'https://LOCALFQDNSERVER/' is invalid since it is already in use","innerError":{"date":"2022-01-31T22:09:58","request-id":"d5c4dfe0-096d-4382-9da0-9559f45e0217","client-request-id":"d5c4dfe0-096d-4382-9da0-9559f45e0217"}}}
We have an apache web server (version httpd-2.2.22-win32-x86-openssl-0.9.8t) with weblogic (version 10.3.2) cluster having 3 nodes. In our load testing, we get session timeout errors in some cases (less than 1%). This was happening, even if we have -1 for session timeout in the web.xmls of weblogic nodes. After days of debugging, we realized that in some cases, the JSESSIOID sent by request is not honored by the response. Fiddler traces show that the RESPONSE has a header named Set-Cookie:JSESSIONID and the value for this is different from the JSESSIONID sent in the request. We get the session expiry page immediately. As already mentioned, this happens only in some rare cases.
When using WeblogicCluster, the requests have session affinity. Thus the requests go to the same node where the initial contact was made. But the issue turned out to be that at high loads, the nodes were not responding. Thus the requests go to the other nodes. This is the default behavior with WeblogicCluster. Since we do not have session replication and failover enabled, any request that goes to the secondary nodes would give us session timeout error.
One solution to this was to start supporting session replication and failover in weblogic. But we did not want that as the impact was high.
These were the configuration changes that fixed this issue
In httpd.conf
ConnectTimeoutSecs 50 (default is 10)
ConnectRetrySecs 5 (default is 2)
WLSocketTimeoutSecs 10 (default is 2)
WLIOTimeoutSecs 18000 (default is 300)
Idempotent OFF (default is ON)
The first 2 changes in ConnectTimeoutSecs and ConnectRetrySecs means that
retry would 10 times (50/5) instead of the default 5 (10/2)
In weblogic nodes
domain --> environment --> servers --> click on the required server -->
tuning--> Accept Backlog:
--> default value is 300. Made it 375.
restart the weblogic nodes and apache
For more details refer
http://docs.oracle.com/cd/E13222_01/wls/docs81/plugins/plugin_params.html
http://docs.oracle.com/cd/E13222_01/wls/docs81/plugins/apache.html . Refer the diagram here for the
I was reading this PDF: http://2010.lucene-eurocon.org/slides/Solr-In-The-Cloud_Mark-Miller.pdf, and there is a section that talks about CloudSolrServer. In particular, this statement is made:
It keeps a list of both live and dead servers. When a request to a server fails, that server is added to the ‘dead’ list, and another ‘live’ server is queried instead.
The ‘dead’ server list is occasionally pinged, and if a server comes back, it is moved back into the ‘live’ list.
This works fine when a SOLR instance or the machine crashes, but for normal maintenance it would be undesirable because requests in progress would be lost. Typically with a normal load balancer, there's a way to shut off traffic to a box, and then normal shutdown can proceed at some interval after that.
Since it appears that CloudSolrServer is intended to replace a traditional load balancer in front of a SOLR cluster, I was wondering about graceful shutdown. What is the recommended way to shutdown a SOLR instance while not losing requests, (while using CloudSolrServer)?
If you want to gracefully shutdown an instance, you will need to first remove the corresponding node from ZooKeeper and then shut down the instance. You can remove the node from ZK by using "DELETEREPLICA" command:
/admin/collections?action=DELETEREPLICA&collection=collection&shard=shard&replica=replica
See more in Solr Collections API documentation
once the ephemeral node is removed from ZooKeeper, CloudSolrServer will stop sending requests to it.
I followed the installation guide for an Apache Web Policy Agent, but it always results in an endless redirect loop between web and application server. Firefox says "The page isn't redirecting properly" and Chrome thinks that "This webpage has a redirect loop". The setup is an Apache 2 on port 80 with a small demo app and a Web Policy Agent, and a Tomcat 7 server on port 8080 with an OpenAM server (the former OpenSSO from Sun):
App URL http://hostname.example.com:80/ (App and Agent, running on Apache 2.2.16)
OpenAM Server URL http://hostname.example.com:8080/openam (running on Tomcat 7.0.12)
The Live HTTP Header Firefox plugin shows that the policy agent and the OpenAM server (i.e. the Apache and Tomcat servers) redirect to each other, although the server sets the SSO Token Cookie correctly. The name of the SSO Token Cookie has the default value "iPlanetDirectoryPro". Any idea how to solve the problem?
After a whole week I finally figured it out, with the help of Stackoverflow and the OpenAM Mailing list. There were two main problems: missing log files and missing cookie domains. Installing the OpenAM server and the Web Policy Agent is difficult, there are a lot of log files and many different configuration options. If you select the wrong options, it won't work. It is impossible to make it work without knowing what is going on, which can only be determined by a suitable log file.
Missing Log for Web Policy Agent : The log level must be set in the "Java properties" files. There are two "Java Properties" files for the Web Policy Agent, OpenSSOAgentBootstrap.properties and OpenSSOAgentConfiguration.properties. The log and debug level which is named com.sun.identity.agents.config.debug.level can and must be defined in both (!) files, and it should be set to the high level, all:4 or all:5. The format is important. Even if you do this, the AgentConfiguration.properties file is only used when the agent is not working in centralized config mode. The profile must be set to "local".
Missing Cookie Domain: Do not forget to enter the right Cookie Domain during the setup of the OpenAM server in the beginning, or add it afterwards if it is missing. On the OpenAM server, go to Configuration > System > Platform and change the Cookie Domain Value to your domain (for instace .example.com) if it is missing. Otherwise the browser will lose your cookie during the redirect process. Somehow I had an empty entry for the cookie domain at the OpenAM server, I guess a forgot a dot (example.com instead of .example.com) so that it was invalid (or something like that).
This troubleshooting site was helpful to locate the problem.
#0x4a6f4672, Your post was absolutely helpful . Some more to add to your answer. The following changes is what i had to do in the config to make it running, at-least getting it running it for alfresco.
com.sun.identity.agents.config.user.mapping.mode=USER_ID(Dont use HTTP_HEADER)
com.sun.identity.agents.config.user.attribute.name=uid
com.sun.identity.agents.config.user.principal=true(Dont use false)
com.sun.identity.agents.config.user.token=SsoUserHeader(Keep it as per what is specified in you application- in my case alfresco)
Now you are not running the Agent in centralised mode but in local mode the setting which is specified for profile attribute can be only set via property file so add the following.
com.sun.identity.agents.config.profile.attribute.fetch.mode=HTTP_HEADER
com.sun.identity.agents.config.profile.attribute.mapping[uid]=SsoUserHeader(whatever you want the header to come in browser as)
As told by 0x4a6f4672, it is difficult to debug and unless you are in local mode , so switch to local mode immediately and start tracing the logs and make the property changes accordingly.
Suppose I have 3 webservers behind a reverse proxy/loadbalancer. Currently I use Apache 2 but don't mind switching.
Those webservers are not equal. They have different sets of local data meaning that different kinds of requests will be handled more efficiently on specific webserver.
I want to plug some custom routing logic into my loadbalancer, that would on start of a new session select a webserver for it, then tie that session to that node until it breaks down (sticky session).
For example: a new user comes to the webapp, load balancer runs my custom code somehow, this code finds out information about that user and makes a conclusion that this user should be handled on node1. The balancer proxies that user to node1 and sticks it to it. When another user comes, custom routing logic suggests node3, to which loadbalancer then proxies this user.
Is it possible to do with Apache 2? If not, what proxy/loadbalancer would give me that ability?
Thank you.
Apache mode_proxy_balancer has stickyness if you want to stay on apache.
Otherwise, have a look at Haproxy. The first line of the abstract states: "route HTTP requests depending on statically assigned cookies ;"
You just have to set a cookie to know which backend you want the user to be stuck to.