CAS Clustering Not Working
Scott Battaglia
scott.battaglia at gmail.com
Tue Oct 16 13:51:50 EDT 2007
I'm not too familiar with JBossCache so I'm going to be pretty much zero
help here :-). However as you guys figure things out, if you think any of
this troubleshooting/configuration could be helpful, please feel free to add
it to the wiki to help other people out (and to make sure you don't need to
answer the question again ;-)).
Thanks!
-Scott
On 10/16/07, Andrew R Feller <afelle1 at lsu.edu> wrote:
>
> Cliff,
>
>
>
> To be honest, I never was able to get multicasting setup properly for us,
> so we went with unicast, which worked out well considering our off-campus
> sites. However, I do remember a similar scenario when I was trying to work
> with multicasting.
>
>
>
> The mcastTTL is the amount of time the heartbeat ping has to check to see
> if a cluster member is alive. If both of your machines were on the same
> subnet, then you would be able to set this to 1 to limit the amount of time
> it takes. However, there is a separate configuration for how long to wait
> between heartbeat checks if I recall correctly. By seeing your member
> dropped and then added in a split second, I wonder if it is an issue of the
> delay between heartbeat checks or something else. I believe this is
> controlled by the <PING> entity's timeout attribute. Anyone know?
>
>
>
> Hope that helps,
>
>
>
> Andrew R Feller, Analyst
>
> Subversion Administrator
>
> University Information Systems
>
> Louisiana State University
>
> afelle1 at lsu.edu
>
> (office) 225.578.3737
> ------------------------------
>
> *From:* cas-bounces at tp.its.yale.edu [mailto:cas-bounces at tp.its.yale.edu] *On
> Behalf Of *Clifford Bryant
> *Sent:* Tuesday, October 16, 2007 9:37 AM
> *To:* Yale CAS mailing list
> *Subject:* RE: CAS Clustering Not Working
>
>
>
> Hi Andrew,
>
>
>
> Thanks for the advice. Actually, the CAS clustering appears to be working
> now. I sent out an email at 3:00 AM this morning. I set the
> tcpListenAddress to the IP address of each of the servers. It was a
> beautiful thing to see that "memberAdded" message in the logJ
>
>
>
> I am still getting the memberAdded/memberDisappeared sequence in the log.
> It may have something to do with the multicast settings. The mcastTTL is
> currently set at 250 ms. It looks like the member gets added back in right
> away (within the same second).
>
>
>
> Cliff Bryant
>
>
>
> Here is the relevant portion of the log file:
>
>
>
> Oct 16, 2007 2:25:53 AM org.apache.catalina.cluster.tcp.SimpleTcpClusterstart
>
> INFO: Cluster is about to start
>
> Oct 16, 2007 2:25:54 AM
> org.apache.catalina.cluster.tcp.ReplicationTransmitter start
>
> INFO: Start ClusterSender at cluster
> Standalone:type=Cluster,host=localhost with name
> Standalone:type=ClusterSender,host=localhost
>
> Oct 16, 2007 2:25:54 AM org.apache.catalina.cluster.mcast.McastServiceImplsetupSocket
>
> INFO: Setting cluster mcast TTL to 250
>
> Oct 16, 2007 2:25:54 AM org.apache.catalina.cluster.mcast.McastServicestart
>
> INFO: Sleeping for 2000 milliseconds to establish cluster membership
>
> Oct 16, 2007 2:25:54 AM org.apache.catalina.cluster.tcp.SimpleTcpClustermemberAdded
>
> INFO: Replication member added:
> org.apache.catalina.cluster.mcast.McastMember
> [tcp://10.10.10.96:4001,catalina,10.10.10.96,4001, alive=54829]
>
> Oct 16, 2007 2:25:56 AM org.apache.catalina.cluster.mcast.McastServiceregisterMBean
>
> INFO: membership mbean registered
> (Standalone:type=ClusterMembership,host=localhost)
>
> Oct 16, 2007 2:25:58 AM org.apache.catalina.cluster.session.DeltaManagerstart
>
> INFO: Register manager /cas to cluster element Host with name localhost
>
> Oct 16, 2007 2:25:58 AM org.apache.catalina.cluster.session.DeltaManagerstart
>
> INFO: Starting clustering manager at /cas
>
> Oct 16, 2007 2:25:58 AM org.apache.catalina.cluster.session.DeltaManagergetAllClusterSessions
>
> WARNING: Manager [/cas], requesting session state from
> org.apache.catalina.cluster.mcast.McastMember
> [tcp://10.10.10.96:4001,catalina,10.10.10.96,4001, alive=57847]. This
> operation will timeout if no session state has been received within 60
> seconds.
>
> Oct 16, 2007 2:25:58 AM org.apache.catalina.cluster.session.DeltaManagerwaitForSendAllSessions
>
> INFO: Manager [/cas]; session state send at 10/16/07 2:25 AM received in
> 154 ms.
>
> 2007-10-16 02:26:01,011 INFO [org.jasig.cas.util.JBossCacheFactoryBean] -
> <Starting TreeCache service.>
>
>
>
> -------------------------------------------------------
>
> GMS: address is 127.0.0.1:32790
>
> -------------------------------------------------------
>
>
>
> And then later on ….
>
>
>
> Oct 16, 2007 2:26:31 AM org.apache.catalina.core.ApplicationContext log
>
> INFO: ContextListener: contextInitialized()
>
> Oct 16, 2007 2:26:31 AM org.apache.catalina.core.ApplicationContext log
>
> INFO: SessionListener: contextInitialized()
>
> Oct 16, 2007 2:26:32 AM org.apache.coyote.http11.Http11BaseProtocol start
>
> INFO: Starting Coyote HTTP/1.1 on http-8080
>
> Oct 16, 2007 2:26:32 AM org.apache.coyote.http11.Http11BaseProtocol start
>
> INFO: Starting Coyote HTTP/1.1 on http-8443
>
> Oct 16, 2007 2:26:32 AM org.apache.jk.common.ChannelSocket init
>
> INFO: JK: ajp13 listening on /0.0.0.0:8009
>
> Oct 16, 2007 2:26:32 AM org.apache.jk.server.JkMain start
>
> INFO: Jk running ID=0 time=0/0 config=null
>
> Oct 16, 2007 2:26:32 AM org.apache.catalina.storeconfig.StoreLoader load
>
> INFO: Find registry server-registry.xml at classpath resource
>
> Oct 16, 2007 2:26:33 AM org.apache.catalina.startup.Catalina start
>
> INFO: Server startup in 39544 ms
>
> Oct 16, 2007 2:29:23 AM org.apache.catalina.cluster.tcp.SimpleTcpClustermemberDisappeared
>
> INFO: Received member disappeared:
> org.apache.catalina.cluster.mcast.McastMember
> [tcp://10.10.10.96:4001,catalina,10.10.10.96,4001, alive=248388]
>
> Oct 16, 2007 2:29:23 AM org.apache.catalina.cluster.tcp.SimpleTcpClustermemberAdded
>
> INFO: Replication member added:
> org.apache.catalina.cluster.mcast.McastMember
> [tcp://10.10.10.96:4001,catalina,10.10.10.96,4001, alive=248892]
>
> Oct 16, 2007 2:31:28 AM org.apache.catalina.cluster.tcp.SimpleTcpClustermemberDisappeared
>
> INFO: Received member disappeared:
> org.apache.catalina.cluster.mcast.McastMember
> [tcp://10.10.10.96:4001,catalina,10.10.10.96,4001, alive=382262]
>
> Oct 16, 2007 2:31:28 AM org.apache.catalina.cluster.tcp.SimpleTcpClustermemberAdded
>
> INFO: Replication member added:
> org.apache.catalina.cluster.mcast.McastMember
> [tcp://10.10.10.96:4001,catalina,10.10.10.96,4001, alive=382765]
>
> Oct 16, 2007 2:32:14 AM org.apache.catalina.cluster.tcp.SimpleTcpClustermemberDisappeared
>
> INFO: Received member disappeared:
> org.apache.catalina.cluster.mcast.McastMember
> [tcp://10.10.10.96:4001,catalina,10.10.10.96,4001, alive=427479]
>
> Oct 16, 2007 2:32:14 AM org.apache.catalina.cluster.tcp.SimpleTcpClustermemberAdded
>
> INFO: Replication member added:
> org.apache.catalina.cluster.mcast.McastMember
> [tcp://10.10.10.96:4001,catalina,10.10.10.96,4001, alive=427982]
>
> Oct 16, 2007 2:33:12 AM org.apache.catalina.cluster.tcp.SimpleTcpClustermemberDisappeared
>
> INFO: Received member disappeared:
> org.apache.catalina.cluster.mcast.McastMember
> [tcp://10.10.10.96:4001,catalina,10.10.10.96,4001, alive=491458]
>
> Oct 16, 2007 2:33:12 AM org.apache.catalina.cluster.tcp.SimpleTcpClustermemberAdded
>
> INFO: Replication member added:
> org.apache.catalina.cluster.mcast.McastMember
> [tcp://10.10.10.96:4001,catalina,10.10.10.96,4001, alive=491959]
>
> Oct 16, 2007 2:34:09 AM org.apache.catalina.cluster.tcp.SimpleTcpClustermemberDisappeared
>
> INFO: Received member disappeared:
> org.apache.catalina.cluster.mcast.McastMember
> [tcp://10.10.10.96:4001,catalina,10.10.10.96,4001, alive=545128]
>
> 2007-10-16 02:34:12,132 INFO [
> org.jasig.cas.web.flow.AutomaticCookiePathSetterAction] - <Setting
> ContextPath for cookies to: /cas>
>
>
> ------------------------------
>
> *From:* cas-bounces at tp.its.yale.edu [mailto:cas-bounces at tp.its.yale.edu] *On
> Behalf Of *Andrew R Feller
> *Sent:* Tuesday, October 16, 2007 10:15 AM
> *To:* Yale CAS mailing list
> *Subject:* RE: CAS Clustering Not Working
>
>
>
> Clifford,
>
>
>
> You are correct, JBoss Cache is to replicate CAS' ticket registry. This
> ticket registry along with Tomcat's session information must be replicated
> amongst all of the clustered machines. Excuse the obligatory "did you"
> checklist:
>
>
>
> - Configured Tomcat's conf/server.xml for session replication
> (different settings than JBoss)
> - Configured firewall to accept connections for both Tomcat and
> JBoss session replication
> - Modified applicationContext for ticket uniqueness and configured
> cas.properties file with servers' names
>
>
>
> If you could, configure Log4J to log debug messages for
> org.jasig.cas.ticket.registry and org.apache.catalina.cluster. This might
> tell you why Apache / JBoss hasn't found the other member.
>
>
>
> Andrew R Feller, Analyst
>
> Subversion Administrator
>
> University Information Systems
>
> Louisiana State University
>
> afelle1 at lsu.edu
>
> (office) 225.578.3737
> ------------------------------
>
> *From:* cas-bounces at tp.its.yale.edu [mailto:cas-bounces at tp.its.yale.edu] *On
> Behalf Of *Clifford Bryant
> *Sent:* Tuesday, October 16, 2007 7:36 AM
> *To:* Yale CAS mailing list
> *Subject:* RE: CAS Clustering Not Working
>
>
>
> The production environment has 4 Apache/Tomcat servers fronted by a BigIP
> load balancer.
>
>
>
> To test out the concept, I have 2 (virtual) Linux servers each running
> Tomcat, with Apache load balancer on one of them.
>
>
>
> I thought that the JBoss stuff was for the Ticket Cache Replication?
>
>
> ------------------------------
>
> *From:* cas-bounces at tp.its.yale.edu [mailto:cas-bounces at tp.its.yale.edu] *On
> Behalf Of *Andrew R Feller
> *Sent:* Tuesday, October 16, 2007 8:17 AM
> *To:* Yale CAS mailing list
> *Subject:* RE: CAS Clustering Not Working
>
>
>
> Clifford,
>
>
>
> What is your intended scenario? Have you setup two instances of CAS on a
> single machine or are they on different machines? It appears as though you
> have JBoss configured for localhost (127.0.0.1:32789), which makes me
> curious.
>
>
>
> Andrew R Feller, Analyst
>
> Subversion Administrator
>
> University Information Systems
>
> Louisiana State University
>
> afelle1 at lsu.edu
>
> (office) 225.578.3737
> ------------------------------
>
> *From:* cas-bounces at tp.its.yale.edu [mailto:cas-bounces at tp.its.yale.edu] *On
> Behalf Of *Clifford Bryant
> *Sent:* Monday, October 15, 2007 12:58 PM
> *To:* cas at tp.its.yale.edu
> *Subject:* CAS Clustering Not Working
>
>
>
> Hi,
>
>
>
> I followed the instructions in the CAS clustering link. The multicast
> ping is working. But, the CAS clustering is not working. If I shut down
> the first server, and browse to the second server, then I am prompted to
> login again to the second server.
>
>
>
> This is the log from Catalina.out.
>
>
>
> Oct 15, 2007 12:44:24 PM org.apache.catalina.cluster.tcp.SimpleTcpClusterstart
>
> INFO: Cluster is about to start
>
> Oct 15, 2007 12:44:24 PM
> org.apache.catalina.cluster.tcp.ReplicationTransmitter start
>
> INFO: Start ClusterSender at cluster Catalina:type=Cluster,host=localhost
> with name Catalina:type=ClusterSender,host=localhost
>
> Oct 15, 2007 12:44:24 PM
> org.apache.catalina.cluster.mcast.McastServiceImpl setupSocket
>
> INFO: Setting cluster mcast TTL to 1
>
> Oct 15, 2007 12:44:24 PM org.apache.catalina.cluster.mcast.McastServicestart
>
> INFO: Sleeping for 2000 milliseconds to establish cluster membership
>
> Oct 15, 2007 12:44:26 PM org.apache.catalina.cluster.mcast.McastServiceregisterMBean
>
> INFO: membership mbean registered
> (Catalina:type=ClusterMembership,host=localhost)
>
> Oct 15, 2007 12:44:29 PM org.apache.catalina.cluster.session.DeltaManagerstart
>
> INFO: Register manager /cas to cluster element Host with name localhost
>
> Oct 15, 2007 12:44:29 PM org.apache.catalina.cluster.session.DeltaManagerstart
>
> INFO: Starting clustering manager at /cas
>
> Oct 15, 2007 12:44:29 PM org.apache.catalina.cluster.session.DeltaManagergetAllClusterSessions
>
> INFO: Manager [/cas]: skipping state transfer. No members active in
> cluster group.
>
> 2007-10-15 12:44:32,075 INFO [org.jasig.cas.util.JBossCacheFactoryBean] -
> <Starting TreeCache service.>
>
>
>
> -------------------------------------------------------
>
> GMS: address is 127.0.0.1:32789
>
>
>
> Cliff Bryant
>
>
>
>
>
>
>
> This e-mail and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed. This communication may contain information that is protected from disclosure by applicable law. If you are not the intended recipient, or the employee or agent responsible for delivering this communication to the intended recipient, be advised that you have received this e-mail in error and any use, dissemination, forwarding, printing or copying of this e-mail is strictly prohibited. If you believe that you have received this e-mail in error, please immediately notify Edgewater Technology by telephone at (781) 246-3343 and delete the communication from all e-mail files.
>
>
>
>
>
>
>
> This e-mail and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed. This communication may contain information that is protected from disclosure by applicable law. If you are not the intended recipient, or the employee or agent responsible for delivering this communication to the intended recipient, be advised that you have received this e-mail in error and any use, dissemination, forwarding, printing or copying of this e-mail is strictly prohibited. If you believe that you have received this e-mail in error, please immediately notify Edgewater Technology by telephone at (781) 246-3343 and delete the communication from all e-mail files.
>
>
>
>
>
>
>
> This e-mail and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed. This communication may contain information that is protected from disclosure by applicable law. If you are not the intended recipient, or the employee or agent responsible for delivering this communication to the intended recipient, be advised that you have received this e-mail in error and any use, dissemination, forwarding, printing or copying of this e-mail is strictly prohibited. If you believe that you have received this e-mail in error, please immediately notify Edgewater Technology by telephone at (781) 246-3343 and delete the communication from all e-mail files.
>
>
>
>
> _______________________________________________
> Yale CAS mailing list
> cas at tp.its.yale.edu
> http://tp.its.yale.edu/mailman/listinfo/cas
>
>
--
-Scott Battaglia
LinkedIn: http://www.linkedin.com/in/scottbattaglia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://tp.its.yale.edu/pipermail/cas/attachments/20071016/d9e1f358/attachment.html
More information about the cas
mailing list