MemCacheTicketRegistry and memcached/repcached failure
Scott Battaglia
scott.battaglia at gmail.com
Sun Jan 11 22:19:29 EST 2009
In general there may be a short window where it doesn't fail over correctly
(the client basically checks the servers to see if the are up). If we're
not correctly capturing that error and at least just returning null, we
should.
-Scott
-Scott Battaglia
PGP Public Key Id: 0x383733AA
LinkedIn: http://www.linkedin.com/in/scottbattaglia
On Fri, Jan 9, 2009 at 1:40 AM, Patrick Hennessy <lists-cas at dtcc.edu> wrote:
>
> I have a couple Tomcat instances on a pair of servers and another pair
> running the repcache patched version of memcached.
>
> I thought that running the two memcache instances would be a little more
> fault tolerant. But it seems when one of them goes down, Tomcat will
> throw errors. I'm wondering if I have some setting misconfigured. The
> errors will show in the window where someone is trying to get a ticket
> for an application.
>
> Below is from my catalina.log..
>
>
> 2009-01-09 01:12:46.000 INFO net.spy.memcached.MemcachedConnection:
> Reconnecting due to exception on {QA sa=hostname/ip_address:11211,
> #Rops=3, #Wops=0, #iq=0,
> topRop=net.spy.memcached.protocol.ascii.StoreOperationImpl at 128b30e,
> topWop=null, toWrite=72, interested=5}
> java.io.IOException: Broken pipe
> at sun.nio.ch.FileDispatcher.write0(Native Method)
> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)
> at sun.nio.ch.IOUtil.write(IOUtil.java:75)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
> at
>
> net.spy.memcached.protocol.TCPMemcachedNodeImpl.writeSome(TCPMemcachedNodeImpl.java:360)
> at
>
> net.spy.memcached.MemcachedConnection.handleWrites(MemcachedConnection.java:283)
> at
>
> net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:265)
> at
>
> net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:182)
> at net.spy.memcached.MemcachedClient.run(MemcachedClient.java:1259)
> 2009-01-09 01:12:46.002 WARN net.spy.memcached.MemcachedConnection:
> Closing, and reopening {QA sa=hostname/ip_address:11211, #Rops=3,
> #Wops=0, #iq=0,
> topRop=net.spy.memcached.protocol.ascii.StoreOperationImpl at 128b30e,
> topWop=null, toWrite=72, interested=5}, attempt 0.
> 2009-01-09 01:12:46.017 WARN
> net.spy.memcached.protocol.ascii.AsciiMemcachedNodeImpl: Discarding
> partially completed op:
> net.spy.memcached.protocol.ascii.StoreOperationImpl at 128b30e
> 2009-01-09 01:12:46.017 WARN
> net.spy.memcached.protocol.ascii.AsciiMemcachedNodeImpl: Discarding
> partially completed op:
> net.spy.memcached.protocol.ascii.GetOperationImpl at 16ba008
> 2009-01-09 01:12:46.018 WARN
> net.spy.memcached.protocol.ascii.AsciiMemcachedNodeImpl: Discarding
> partially completed op:
> net.spy.memcached.protocol.ascii.GetOperationImpl at d78cb8
> 2009-01-09 01:12:46,055 ERROR
> [org.apache.catalina.core.ContainerBase.[Catalina].[c-cas-02.dtcc.edu
> ].[/cas].[cas]]
> - <Servlet.service() for servlet cas threw exception>
> java.lang.RuntimeException: Cancelled
> at
>
> net.spy.memcached.MemcachedClient$OperationFuture.get(MemcachedClient.java:1443)
> at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:764)
> at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:784)
> at
>
> org.jasig.cas.ticket.registry.MemCacheTicketRegistry.getTicket(MemCacheTicketRegistry.java:109)
> at
>
> org.jasig.cas.audit.spi.TicketOrCredentialBasedAuditablePrincipalResolver.resolveFromInternal(TicketOrCredentialBasedAuditablePrincipalResolver.java:50)
> at
>
> org.jasig.cas.audit.spi.TicketOrCredentialBasedAuditablePrincipalResolver.resolveFrom(TicketOrCredentialBasedAuditablePrincipalResolver.java:39)
> at
>
> org.inspektr.audit.AuditTrailManagementAspect.handleAuditTrail(AuditTrailManagementAspect.java:116)
> at sun.reflect.GeneratedMethodAccessor44.invoke(Unknown Source)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
>
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:627)
> at
>
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:616)
> at
>
> org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:64)
> at
>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:160)
> at
>
> org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:77)
> at
>
> org.inspektr.statistics.StatisticManagementAspect.handleStatisticGathering(StatisticManagementAspect.java:73)
> at sun.reflect.GeneratedMethodAccessor46.invoke(Unknown Source)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
>
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:627)
> at
>
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:616)
> at
>
> org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:64)
> at
>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:160)
> at
>
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
> at
>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
> at
>
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at $Proxy9.grantServiceTicket(Unknown Source)
> at
>
> org.jasig.cas.web.flow.GenerateServiceTicketAction.doExecute(GenerateServiceTicketAction.java:37)
> at
>
> org.springframework.webflow.action.AbstractAction.execute(AbstractAction.java:192)
> at
>
> org.springframework.webflow.engine.AnnotatedAction.execute(AnnotatedAction.java:146)
> at
>
> org.springframework.webflow.engine.ActionExecutor.execute(ActionExecutor.java:59)
> at
>
> org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:156)
> at org.springframework.webflow.engine.State.enter(State.java:191)
> at
> org.springframework.webflow.engine.Transition.execute(Transition.java:212)
> at
>
> org.springframework.webflow.engine.DecisionState.doEnter(DecisionState.java:54)
> at org.springframework.webflow.engine.State.enter(State.java:191)
> at
> org.springframework.webflow.engine.Transition.execute(Transition.java:212)
> at
>
> org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107)
> at org.springframework.webflow.engine.Flow.onEvent(Flow.java:534)
> at
>
> org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205)
> at
>
> org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161)
> at org.springframework.webflow.engine.State.enter(State.java:191)
> at
> org.springframework.webflow.engine.Transition.execute(Transition.java:212)
> at
>
> org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107)
> at org.springframework.webflow.engine.Flow.onEvent(Flow.java:534)
> at
>
> org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205)
> at
>
> org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161)
> at org.springframework.webflow.engine.State.enter(State.java:191)
> at
> org.springframework.webflow.engine.Transition.execute(Transition.java:212)
> at
>
> org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107)
> at org.springframework.webflow.engine.Flow.onEvent(Flow.java:534)
> at
>
> org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205)
> at
>
> org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161)
> at org.springframework.webflow.engine.State.enter(State.java:191)
> at
> org.springframework.webflow.engine.Transition.execute(Transition.java:212)
> at
>
> org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107)
> at org.springframework.webflow.engine.Flow.onEvent(Flow.java:534)
> at
>
> org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205)
> at
>
> org.springframework.webflow.engine.impl.FlowExecutionImpl.signalEvent(FlowExecutionImpl.java:202)
> at
>
> org.springframework.webflow.executor.FlowExecutorImpl.resume(FlowExecutorImpl.java:222)
> at
>
> org.springframework.webflow.executor.support.FlowRequestHandler.handleFlowRequest(FlowRequestHandler.java:111)
> at
>
> org.springframework.webflow.executor.mvc.FlowController.handleRequestInternal(FlowController.java:165)
> at
>
> org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
> at
>
> org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
> at
>
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:875)
> at
>
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:807)
> at
>
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:571)
> at
>
> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:511)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
> at
>
> org.jasig.cas.web.init.SafeDispatcherServlet.service(SafeDispatcherServlet.java:115)
> at
>
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
> at
>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
> at
>
> org.inspektr.common.web.ClientInfoThreadLocalFilter.doFilterInternal(ClientInfoThreadLocalFilter.java:48)
> at
>
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
> at
>
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
> at
>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
> at
>
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:210)
> at
>
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
> at
>
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525)
> at
>
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> at
>
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
> at
>
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
> at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:870)
> at
>
> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
> at
>
> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
> at
>
> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
> at
>
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:685)
> at java.lang.Thread.run(Thread.java:619)
> 2009-01-09 01:12:46.119 INFO net.spy.memcached.MemcachedConnection:
> Reconnecting {QA sa=hostname/ip_address:11211, #Rops=0, #Wops=0, #iq=0,
> topRop=null, topWop=null, toWrite=0, interested=0}
> 2009-01-09 01:12:46.121 INFO net.spy.memcached.MemcachedConnection:
> Connection state changed for sun.nio.ch.SelectionKeyImpl at b1625c
> 2009-01-09 01:12:46.123 INFO net.spy.memcached.MemcachedConnection:
> Reconnecting due to exception on {QA sa=hostname/ip_address:11211,
> #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0}
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at
>
> net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:250)
> at
>
> net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:182)
> at net.spy.memcached.MemcachedClient.run(MemcachedClient.java:1259)
> 2009-01-09 01:12:46.125 WARN net.spy.memcached.MemcachedConnection:
> Closing, and reopening {QA sa=hostname/ip_address:11211, #Rops=0,
> #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0},
> attempt 1.
> 2009-01-09 01:12:46.328 INFO net.spy.memcached.MemcachedConnection:
> Reconnecting {QA sa=hostname/ip_address:11211, #Rops=0, #Wops=0, #iq=0,
> topRop=null, topWop=null, toWrite=0, interested=0}
> 2009-01-09 01:12:46.329 INFO net.spy.memcached.MemcachedConnection:
> Connection state changed for sun.nio.ch.SelectionKeyImpl at 16df0d6
> 2009-01-09 01:12:46.330 INFO net.spy.memcached.MemcachedConnection:
> Reconnecting due to exception on {QA sa=hostname/ip_address:11211,
> #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0}
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at
>
> net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:250)
> at
>
> net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:182)
> at net.spy.memcached.MemcachedClient.run(MemcachedClient.java:1259)
>
>
> The four lines or so repeat themselves until I restart memcached. I can
> grab some of my config files and whatnot, but I was curious what
> experiences other people had with this.
>
> Thanks in advance,
>
> Pat
>
> --
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> Patrick Hennessy (path at dtcc.edu)
>
> Senior Systems Specialist
> Division of Information and Educational Technology
> Delaware Technical and Community College
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
>
> _______________________________________________
> Yale CAS mailing list
> cas at tp.its.yale.edu
> http://tp.its.yale.edu/mailman/listinfo/cas
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://tp.its.yale.edu/pipermail/cas/attachments/20090111/34f5b6b2/attachment.html
More information about the cas
mailing list