History | Log In     View a printable version of the current page.  
   HOME       BROWSE PROJECT       FIND ISSUES   
    QUICK SEARCH:  
Hyperic HQ 3.2.5-EE Maintenance Release is Now Available
Issue Details (XML | Word | Printable)

Key: HHQ-1961
Type: Bug Bug
Status: Closed Closed
Resolution: Fixed
Priority: Critical Critical
Assignee: Jon Travis
Reporter: Kashyap Parikh
Votes: 0
Watchers: 0
Operations

If you were logged in you would be able to see more operations.
Hyperic HQ

Java level deadlock on ZeventProcessor

Created: 11/Apr/08 08:29 AM   Updated: 16/Apr/08 02:25 PM
Component/s: Server
Affects Version/s: 3.2.2
Fix Version/s: 3.2.3

File Attachments: 1. GZip Archive server.out.gz (37 kb)
Environment: 3.2.2 #650 with mysql backend

Verify By: Kashyap Parikh
Last comment: 25 weeks, 5 days ago
Resolution Date: 15/Apr/08 12:03 PM


 Description  « Hide
After running the server under heavy load for a day I am seeing following java level deadlock in HQ's thread dump.
The server configuration has ~50K classic value change alerts. All of them are disabled and inactive.
Full thread dump is attached. I see another issue in there and will open another bug for it.

Found one Java-level deadlock:
=============================
"Thread-19":
  waiting to lock monitor 0x0000000049d2f718 (object 0x00002aab0df1b4a8, a java.lang.Object),
  which is held by "ZeventProcessor"
"ZeventProcessor":
  waiting to lock monitor 0x0000000049d2fda8 (object 0x00002aab0df1b418, a java.lang.Object),
  which is held by "Thread-19"

Java stack information for the threads listed above:
===================================================
"Thread-19":
        at org.hyperic.hq.zevents.ZeventManager.getDiagnostics(ZeventManager.java:575)
        - waiting to lock <0x00002aab0df1b4a8> (a java.lang.Object)
        - locked <0x00002aab0df1b418> (a java.lang.Object)
        at org.hyperic.hq.zevents.ZeventManager.access$000(ZeventManager.java:65)
        at org.hyperic.hq.zevents.ZeventManager$1.getStatus(ZeventManager.java:168)
        at org.hyperic.hq.common.DiagnosticThread.run(DiagnosticThread.java:115)
        - locked <0x00002aab0a24f9c0> (a java.util.ArrayList)
        at java.lang.Thread.run(Unknown Source)
"ZeventProcessor":
        at org.hyperic.hq.zevents.ZeventManager.getListenerTimeout(ZeventManager.java:219)
        - waiting to lock <0x00002aab0df1b418> (a java.lang.Object)
        at org.hyperic.hq.zevents.ZeventManager.dispatchEvents(ZeventManager.java:546)
        - locked <0x00002aab0df1b4a8> (a java.lang.Object)
        at org.hyperic.hq.zevents.QueueProcessor.processBatch(QueueProcessor.java:79)
        at org.hyperic.hq.zevents.QueueProcessor.run(QueueProcessor.java:61)
        at java.lang.Thread.run(Unknown Source)

Found 1 deadlock.

 All   Comments   Change History      Sort Order: Ascending order - Click to sort in descending order
[ Permlink | « Hide ]
Ryan Morgan - 11/Apr/08 08:46 AM
Mark 3.2.3, assign to Trav.

[ Permlink | « Hide ]
Kashyap Parikh - 14/Apr/08 08:31 AM
Another thread dump with additional details:

Found one Java-level deadlock:
=============================
"RMI TCP Connection(3989)-10.0.0.110":
  waiting to lock monitor 0x00002aabd40acd18 (object 0x00002aab0eedd1a0, a java.lang.Object),
  which is held by "Thread-32"
"Thread-32":
  waiting to lock monitor 0x00002aabce4b8f08 (object 0x00002aab0eedd230, a java.lang.Object),
  which is held by "ZeventProcessor"
"ZeventProcessor":
  waiting to lock monitor 0x00002aabd40acd18 (object 0x00002aab0eedd1a0, a java.lang.Object),
  which is held by "Thread-32"

Java stack information for the threads listed above:
===================================================
"RMI TCP Connection(3989)-10.0.0.110":
        at org.hyperic.hq.zevents.ZeventManager.getInstance(ZeventManager.java:642)
        - waiting to lock <0x00002aab0eedd1a0> (a java.lang.Object)
        at org.hyperic.hq.bizapp.server.session.HQInternalService.getZeventQueueSize(HQInternalService.java:94)
        at sun.reflect.GeneratedMethodAccessor845.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.jboss.mx.interceptor.AttributeDispatcher.invoke(AttributeDispatcher.java:84)
        at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
        at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
        at org.jboss.mx.interceptor.ModelMBeanAttributeInterceptor.invoke(ModelMBeanAttributeInterceptor.java:182)
        at org.jboss.mx.interceptor.PersistenceInterceptor.invoke(PersistenceInterceptor.java:61)
        at org.jboss.mx.server.Invocation.invoke(Invocation.java:74)
        at org.jboss.mx.server.AbstractMBeanInvoker.getAttribute(AbstractMBeanInvoker.java:343)
        at org.jboss.mx.server.MBeanServerImpl.getAttribute(MBeanServerImpl.java:541)
        at sun.reflect.GeneratedMethodAccessor576.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.jboss.jmx.connector.invoker.InvokerAdaptorService.invoke(InvokerAdaptorService.java:251)
        at sun.reflect.GeneratedMethodAccessor574.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
        at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
        at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:118)
        at org.jboss.mx.server.Invocation.invoke(Invocation.java:74)
        at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:127)
        at org.jboss.mx.server.Invocation.invoke(Invocation.java:74)
        at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:245)
        at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
        at org.jboss.invocation.jrmp.server.JRMPProxyFactory.invoke(JRMPProxyFactory.java:164)
        at sun.reflect.GeneratedMethodAccessor573.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
        at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
        at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
        at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:245)
        at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
        at org.jboss.invocation.jrmp.server.JRMPInvoker$MBeanServerAction.invoke(JRMPInvoker.java:805)
        at org.jboss.invocation.jrmp.server.JRMPInvoker.invoke(JRMPInvoker.java:406)
        at sun.reflect.GeneratedMethodAccessor572.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
        at sun.rmi.transport.Transport$1.run(Transport.java:159)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)
"Thread-32":
        at org.hyperic.hq.zevents.ZeventManager.getDiagnostics(ZeventManager.java:575)
        - waiting to lock <0x00002aab0eedd230> (a java.lang.Object)
        - locked <0x00002aab0eedd1a0> (a java.lang.Object)
        at org.hyperic.hq.zevents.ZeventManager.access$000(ZeventManager.java:65)
        at org.hyperic.hq.zevents.ZeventManager$1.getStatus(ZeventManager.java:168)
        at org.hyperic.hq.common.DiagnosticThread.run(DiagnosticThread.java:115)
        - locked <0x00002aab0b0c5dd0> (a java.util.ArrayList)
        at java.lang.Thread.run(Thread.java:619)
"ZeventProcessor":
        at org.hyperic.hq.zevents.ZeventManager.getListenerTimeout(ZeventManager.java:219)
        - waiting to lock <0x00002aab0eedd1a0> (a java.lang.Object)
        at org.hyperic.hq.zevents.ZeventManager.dispatchEvents(ZeventManager.java:546)
        - locked <0x00002aab0eedd230> (a java.lang.Object)
        at org.hyperic.hq.zevents.QueueProcessor.processBatch(QueueProcessor.java:79)
        at org.hyperic.hq.zevents.QueueProcessor.run(QueueProcessor.java:61)
        at java.lang.Thread.run(Thread.java:619)

Found 1 deadlock.
Powered by a free Atlassian JIRA non-profit license for Hyperic HQ. Try JIRA - bug tracking software for your team.
Atlassian JIRA the Professional Issue Tracker. (Enterprise Edition, Version: 3.12.3-#302) - Bug/feature request - Atlassian news - Contact Administrators