JBossCache Ticket Registry performance under load?
Pat Hennessy
lists-cas at dtcc.edu
Sun Mar 9 09:24:34 EDT 2008
So the first message I got was..
2008-03-06 23:51:20,838 ERROR
[org.jasig.cas.ticket.registry.JBossCacheTicketRegistry] -
<org.jboss.cache.ReplicationException: rsp=sender=138.1
23.130.81:32772, retval=null, received=false, suspected=false>
org.jboss.cache.ReplicationException: rsp=sender=138.123.130.81:32772,
retval=null, received=false, suspected=false
I found an old message on the list..
http://tp.its.yale.edu/pipermail/cas/2006-September/003412.html
I'm using our VMWare Infrastructure for our two nodes. I had them on
two different hosts, so I then moved them to the same VMWare host and
everything appeared to be better. So, next week I plan on talking to
the network guy about using multicast on that switch (I had spoken to
him earlier and he thought the settings on it should be fine).
I left one of my casified apps in the browser all night. This app does
a page refresh and uses one of the apache module, so it's been helpful
in finding these problems.
The next day, I found another exception...
2008-03-08 11:02:09,130 ERROR
[org.apache.catalina.core.ContainerBase.[Catalina].[c-cas-02.dtcc.edu].[/cas].[cas]]
- <Servlet.service() for servlet cas threw exception>
org.jboss.cache.lock.TimeoutException: Response timed out:
sender=138.123.130.81:32772, retval=null, received=false, suspected=false
These are happening on either node and it does appear that I get get
tickets and appear validated from both nodes. So the clustering is
working ok, but there seems to be a timing issue.
Did some more searching and saw references on some Japanese site for
some totally different application to the "SyncReplTimeout" value. I
also found some document on Redhat's site about setting up JBoss. So, I
made some adjustments to the different timeouts. I think I read
something somewhere on some JBoss site that some timeouts need to be
shorter than others. I'm no Tomcat or JBoss expert, but I set the below
settings and I think it's been behaving as expected..
<attribute name="InitialStateRetrievalTimeout">15000</attribute>
<attribute name="SyncReplTimeout">20000</attribute>
<attribute name="LockAcquisitionTimeout">25000</attribute>
I also wonder if the JBoss replication stuff is dependent on the system
clock. I noticed one of them was off and had to fiddle with ntp.
My next goal is to load test logins with JMeter. I did try another
program that someone posted on the list, but I thought JMeter looked
better. I haven't exactly gotten that one working yet. But just
hitting the servers with page retrievals doesn't seem to cause any
exceptions.
Pat
On 3/7/08 5:25 PM, Pat Hennessy wrote:
> On 7/23/2007 11:45 AM, Brian Donnelly wrote:
>> Thanks Scott,
>>
>> I've attached my jbossCache.xml config file. It is almost identical to the jbossTestCache.xml configuration included in CAS 3.0.6. I did have to comment out the authentication protocol version tag because it was generating errors.
>>
>> If anyone has any pointers or would be willing to send their JBossCache configuation parameters, I'd be very appreciative.
>>
>
> Brian,
>
> Did you ever find a fix for the org.jboss.cache.ReplicationException
> error you found?
>
> I just setup the jboss replication using the directions on the CAS wiki
> (and the jbossTestCache.xml file). On the dev cluster, I didn't get the
> error. After putting it on our new to be production cluster, I've been
> finding the same error showing up as a RuntimeException with some of our
> test apps. I don't think we putting these services under any real load
> though.
>
> Pat
>
>> Thanks,
>>
>> Brian Donnelly
>> --
>> Brian Donnelly
>> University of Calfornia, Davis
>> Information and Educational Technology
>> Middleware Team
>> (530) 754-5909
>> bdonnelly at ucdavis.edu
>>
>> -----Original Message-----
>> From: Scott Battaglia [mailto:scott.battaglia at gmail.com]
>> Sent: Fri 7/20/2007 6:29 AM
>> To: Brian Donnelly; Yale CAS mailing list
>> Subject: Re: JBossCache Ticket Registry performance under load?
>>
>> Brian,
>>
>> We don't deploy that at Rutgers so I can't comment on that. A few people
>> have deployed it in production without issues. Maybe you can include your
>> configuration file and those who have deployed it successfully can compare
>> it to theirs if they get a minute (hopefully).
>>
>> Thanks
>> -Scott
>>
>> On 7/18/07, Brian Donnelly <bdonnelly at ucdavis.edu> wrote:
>>> Hi all,
>>>
>>> We're getting ready at UC Davis to switch to a JBossCache Clustered
>>> configuration for our CAS installation. I have been load testing two
>>> Redhat EL 5 clustered nodes running CAS 3.0.6 using the default
>>> JBossCache implementation, (UDP multicast.)
>>>
>>> I've been using JMeter to generate ~7 login actions per second. Both
>>> clustered servers perform fine for several hours. Somewhere in the
>>> third hour of testing, I start seeing the following errors in the logs:
>>>
>>> 2007-07-18 13:43:54,813 ERROR
>>> [org.jasig.cas.ticket.registry.JBossCacheTicketRegistry] -
>>> <org.jboss.cache.ReplicationException: rsp=sender=169.237.104.235:53768,
>>> retval=null, received=false, suspected=false>
>>>
>>> and
>>>
>>> 2007-07-18 13:48:33,448 ERROR
>>> [org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/cas].[cas]]
>>> - <Servlet.service() for servlet cas threw exception>
>>>
>>> These start piling up until both servers stop responding to incoming
>>> requests. A restart is required to restore service.
>>>
>>> Has anyone else encountered errors of this type in their testing of the
>>> JBossCache registry?
>>>
>>> Thanks,
>>>
>>> Brian Donnelly
>>> --
>>> Brian Donnelly
>>> University of Calfornia, Davis
>>> Information and Educational Technology
>>> Middleware Team
>>> (530) 754-5909
>>> bdonnelly at ucdavis.edu
>>> _______________________________________________
>>> Yale CAS mailing list
>>> cas at tp.its.yale.edu
>>> http://tp.its.yale.edu/mailman/listinfo/cas
>>>
>>
>>
>
>
--
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Pat Hennessy, RHCE (path at dtcc.edu)
Senior Systems Specialist
Division of Information and Educational Technology
Delaware Technical and Community College
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
More information about the cas
mailing list